Bias and Information of Bayesian Adaptive Testing

Abstract

Monte carlo simulation was used to investigate score bias and information characteristics of Owen's Bayesian adaptive testing strategy and to examine pos sible causes of score bias. Factors investigated in three related studies included effects of an accurate prior θ estimate, effects of item discrimination, and effects of fixed versus variable test length. Data were generated from a three-parameter logistic model for 3,100 simu lees in each of eight data sets, and Bayesian adaptive tests were administered, drawing items from a "per fect" item pool. Results showed that the Bayesian adaptive test yielded unbiased θ estimates and rela tively flat information functions only in the situation in which an accurate prior θ estimate was used. When a constant prior θ estimate was used with a fixed test length, severe bias was observed that varied with item discrimination. A different pattern of bias was ob served with variable test length and a constant prior. Information curves for the constant prior conditions generally became more peaked and asymmetric with increasing item discrimination. In the variable test length condition, the test length required to achieve a specified level of the posterior variance of θ estimates was an increasing function of θ level. These results indicate that θ estimates from Owen's Bayesian adap tive testing method are affected by the prior θ estimate used and that the method does not provide measure ments that are unbiased and equiprecise except when an accurate prior θ estimate is used.