Quantifying the Effects of Chance in Multiple Choice and True/False Tests: Question selection and guessing of answers

Abstract

Multiple choice and true/false tests commonly contain only a small proportion of all the questions that could be asked, so that an examinee may be lucky or unlucky in the selection. Random guessing further reduces test reliability when it is not effectively discouraged. Four measures of test unreliability are described that quantify the effects of question selection and guessing, both separately and together - three chosen for immediacy and one for greater mathematical elegance. Their dependence on test length and on the number of answer options per question is quantified. Through one or both random effects, many multiple choice tests are too unreliable for their purpose as currently conducted. Ways of improving test reliability are discussed.