On the Post Hoc Power in Testing Mean Differences
- 1 June 2005
- journal article
- Published by American Educational Research Association (AERA) in Journal of Educational and Behavioral Statistics
- Vol. 30 (2), 141-167
- https://doi.org/10.3102/10769986030002141
Abstract
Retrospective or post hoc power analysis is recommended by reviewers and editors of many journals. Little literature has been found that gave a serious study of the post hoc power. When the sample size is large, the observed effect size is a good estimator of the true effect size. One would hope that the post hoc power is also a good estimator of the true power. This article studies whether such a power estimator provides valuable information about the true power. Using analytical, numerical, and Monte Carlo approaches, our results show that the estimated power does not provide useful information when the true power is small. It is almost always a biased estimator of the true power. The bias can be negative or positive. Large sample size alone does not guarantee the post hoc power to be a good estimator of the true power. Actually, when the population variance is known, the cumulative distribution function of the post hoc power is solely a function of the population power. This distribution is uniform when the true power equals 0.5 and highly skewed when the true power is near 0 or 1. When the population variance is unknown, the post hoc power behaves essentially the same as when the variance is known.Keywords
This publication has 13 references indexed in Scilit:
- On Determining Replication Probabilities: Comments on Posavac (2002)Understanding Statistics, 2003
- Using p Values to Estimate the Probability of a Statistically Significant ReplicationUnderstanding Statistics, 2002
- Colloquium on Effect Sizes: the Roles of Editors, Textbook Authors, and the Publication ManualEducational and Psychological Measurement, 2001
- The Abuse of PowerThe American Statistician, 2001
- Power analysis and determination of sample size for covariance structure modeling.Psychological Methods, 1996
- Effect sizes and p values: What should be reported and what should be replicated?Psychophysiology, 1996
- Post hoc power analysis.Journal of Applied Psychology, 1994
- An Average Power Criterion for Sample Size EstimationJournal of the Royal Statistical Society: Series D (The Statistician), 1994
- An Introduction to the BootstrapPublished by Springer Nature ,1993
- CONFIDENCE INTERVALS FOR POWER, WITH SPECIAL REFERENCE TO MEDICAL TRIALS*Australian Journal of Statistics, 1972