The Use of Cognitive Testing to Develop and Evaluate CAHPS™ 1.0 Core Survey Items

Abstract

The main goal of the Consumer Assessments of Health Plans Study (CAHPS) is to develop an integrated set of tested, standardized surveys to obtain meaningful information from health plan enrollees about their experiences. The CAHPS project benefits from the complementary strengths of psychometric and cognitive testing. The CAHPS team conducted 150 cognitive interviews across three organizations in different geographic locations using multiple interview methods with different consumer populations. This article explains how cognitive testing was used in the CAHPS survey development process and shares the main findings from the cognitive interviews. A modified report format is more appropriate when asking about specific aspects of plan enrollees' experiences, whereas a rating format is useful for asking about overall assessments. Specifying a longer reference period is preferable to asking about the most recent visit when capturing experiences with care, because some respondents get frustrated when they cannot include experiences other than the most recent visit. Explicit screeners and tailored inapplicable response categories are beneficial in mail questionnaires, so people know that they should not answer questions about which they have no relevant experience. Cognitive testing was integral in the development and refinement of the CAHPS instrument. The cognitive testing findings contributed to an improved instrument that should capture consumers' health care and plan experiences with less response error than one not subjected to such testing. The cognitive testing process and findings can be useful to other researchers with similar survey development goals.