Measures of Program Effectiveness Based on Retrospective Pretest Data: Are All Created Equal?

Abstract

This study analyzed data from four evaluation designs incorporating the retrospective pretest (i.e., thentest), analyzing the effects of self-report pretesting and post-program survey format on a set of self-report measures. Validity of self-report data was assessed by comparing the criterion validity of then ratings to established benchmarks and by including a control condition. This study found that designs incorporating separate posttest and thentest surveys yielded the most comparable levels of criterion validity for then ratings and the least biased measures of program effectiveness. Conversely, designs that incorporated a single post-program survey, with adjacent post and then items, yielded the least comparable levels of criterion validity for then ratings and the most biased measures of program effectiveness. This study also found a slight interaction between the effects of self-report pretesting and post-program survey format. Implications for program evaluation are discussed.