Running experiments on Amazon Mechanical Turk
Top Cited Papers
Open Access
- 1 August 2010
- journal article
- research article
- Published by Cambridge University Press (CUP) in Judgment and Decision Making
- Vol. 5 (5), 411-419
- https://doi.org/10.1017/s1930297500002205
Abstract
Although Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by subjects recruited from online labor markets. We address these potential concerns by presenting new demographic data about the Mechanical Turk subject population, reviewing the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects, and comparing the magnitude of effects obtained using Mechanical Turk and traditional subject pools. We further discuss some additional benefits such as the possibility of longitudinal, cross cultural and prescreening designs, and offer some advice on how to best manage a common subject pool.Keywords
This publication has 16 references indexed in Scilit:
- On the conjunction fallacy in probability judgment: New experimental evidence regarding LindaGames and Economic Behavior, 2009
- Whatever Happens in the Laboratory Stays in the Laboratory: The Prevalence and Prevention of Participant CrosstalkPersonality and Social Psychology Bulletin, 2009
- Measuring Numeracy without a Math Test: Development of the Subjective Numeracy ScaleMedical Decision Making, 2007
- Should We Trust Web-Based Studies? A Comparative Analysis of Six Preconceptions About Internet Questionnaires.American Psychologist, 2004
- General Performance on a Numeracy Scale among Highly Educated SamplesMedical Decision Making, 2001
- Outcome bias in decision evaluation.Journal of Personality and Social Psychology, 1988
- Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment.Psychological Review, 1983
- The Framing of Decisions and the Psychology of ChoiceScience, 1981
- Interpersonal expectancy effects: the first 345 studiesBehavioral and Brain Sciences, 1978
- On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications.American Psychologist, 1962