The Psychology of Replication and Replication in Psychology
- 7 November 2012
- journal article
- Published by SAGE Publications in Perspectives on Psychological Science
- Vol. 7 (6), 585-594
- https://doi.org/10.1177/1745691612459520
Abstract
Like other scientists, psychologists believe experimental replication to be the final arbiter for determining the validity of an empirical finding. Reports in psychology journals often attempt to prove the validity of a hypothesis or theory with multiple experiments that replicate a finding. Unfortunately, these efforts are sometimes misguided because in a field like experimental psychology, ever more successful replication does not necessarily ensure the validity of an empirical finding. When psychological experiments are analyzed with statistics, the rules of probability dictate that random samples should sometimes be selected that do not reject the null hypothesis, even if an effect is real. As a result, it is possible for a set of experiments to have too many successful replications. When there are too many successful replications for a given set of experiments, a skeptical scientist should be suspicious that null or negative findings have been suppressed, the experiments were run improperly, or the experiments were analyzed improperly. This article describes the implications of this observation and demonstrates how to test for too much successful replication by using a set of experiments from a recent research paper.Keywords
This publication has 23 references indexed in Scilit:
- Evidence that publication bias contaminated studies relating social class and unethical behaviorProceedings of the National Academy of Sciences, 2012
- Measuring the Prevalence of Questionable Research Practices With Incentives for Truth TellingPsychological Science, 2012
- Too good to be true: Publication bias in two prominent studies from experimental psychologyPsychonomic Bulletin & Review, 2012
- The Same Old New Look: Publication Bias in a Study of Wishful SeeingI-Perception, 2012
- Students with the initial “A” don’t get better gradesJournal of Research in Personality, 2011
- Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect.Journal of Personality and Social Psychology, 2011
- Baseball players with the initial “K” do not strike out more oftenJournal of Applied Statistics, 2010
- Why Most Discovered True Associations Are InflatedEpidemiology, 2008
- On the Post Hoc Power in Testing Mean DifferencesJournal of Educational and Behavioral Statistics, 2005
- The earth is round (p < .05).American Psychologist, 1994