Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments
Top Cited Papers
- 25 July 2012
- journal article
- research article
- Published by Springer Science and Business Media LLC in Psychonomic Bulletin & Review
- Vol. 19 (5), 847-857
- https://doi.org/10.3758/s13423-012-0296-9
Abstract
With the increasing sophistication and ubiquity of the Internet, behavioral research is on the cusp of a revolution that will do for population sampling what the computer did for stimulus control and measurement. It remains a common assumption, however, that data from self-selected Web samples must involve a trade-off between participant numbers and data quality. Concerns about data quality are heightened for performance-based cognitive and perceptual measures, particularly those that are timed or that involve complex stimuli. In experiments run with uncompensated, anonymous participants whose motivation for participation is unknown, reduced conscientiousness or lack of focus could produce results that would be difficult to interpret due to decreased overall performance, increased variability of performance, or increased measurement noise. Here, we addressed the question of data quality across a range of cognitive and perceptual tests. For three key performance metrics—mean performance, performance variance, and internal reliability—the results from self-selected Web samples did not differ systematically from those obtained from traditionally recruited and/or lab-tested samples. These findings demonstrate that collecting data from uncompensated, anonymous, unsupervised, self-selected participants need not reduce data quality, even for demanding cognitive and perceptual experiments.Keywords
This publication has 38 references indexed in Scilit:
- Amazon's Mechanical TurkPerspectives on Psychological Science, 2011
- A new selective developmental deficit: Impaired object recognition with normal face recognitionCortex, 2010
- Human face recognition ability is specific and highly heritableProceedings of the National Academy of Sciences, 2010
- A Web-Based Study of Personality, Psychopathology and Substance Use in Twin, Other Relative and Relationship PairsTwin Research and Human Genetics, 2009
- Internet Cognitive Testing of Large Samples Needed in Genetic ResearchTwin Research and Human Genetics, 2007
- Implementing a Five-Factor Personality Inventory for Use on the InternetEuropean Journal of Psychological Assessment, 2005
- Verbal Paired Associates tests limits on validity and reliabilityArchives of Clinical Neuropsychology, 2002
- Comparing the results of laboratory and World-Wide Web samples on the determinants of female attractivenessBehavior Research Methods, Instruments & Computers, 1997
- A power primer.Psychological Bulletin, 1992
- Coefficient alpha and the internal structure of testsPsychometrika, 1951