Reducing Overconfidence in the Interval Judgments of Experts
- 1 March 2010
- journal article
- Published by Wiley in Risk Analysis
- Vol. 30 (3), 512-523
- https://doi.org/10.1111/j.1539-6924.2009.01337.x
Abstract
Elicitation of expert opinion is important for risk analysis when only limited data are available. Expert opinion is often elicited in the form of subjective confidence intervals; however, these are prone to substantial overconfidence. We investigated the influence of elicitation question format, in particular the number of steps in the elicitation procedure. In a 3-point elicitation procedure, an expert is asked for a lower limit, upper limit, and best guess, the two limits creating an interval of some assigned confidence level (e.g., 80%). In our 4-step interval elicitation procedure, experts were also asked for a realistic lower limit, upper limit, and best guess, but no confidence level was assigned; the fourth step was to rate their anticipated confidence in the interval produced. In our three studies, experts made interval predictions of rates of infectious diseases (Study 1, n = 21 and Study 2, n = 24: epidemiologists and public health experts), or marine invertebrate populations (Study 3, n = 34: ecologists and biologists). We combined the results from our studies using meta-analysis, which found average overconfidence of 11.9%, 95% CI [3.5, 20.3] (a hit rate of 68.1% for 80% intervals)-a substantial decrease in overconfidence compared with previous studies. Studies 2 and 3 suggest that the 4-step procedure is more likely to reduce overconfidence than the 3-point procedure (Cohen's d = 0.61, [0.04, 1.18]).Keywords
This publication has 14 references indexed in Scilit:
- Overconfidence in interval estimates: What does expertise buy you?Organizational Behavior and Human Decision Processes, 2008
- Effects of amount of information on judgment accuracy and confidenceOrganizational Behavior and Human Decision Processes, 2008
- When 90% confidence intervals are 50% certain: on the credibility of credible intervalsApplied Cognitive Psychology, 2005
- Overconfidence in Interval Estimates.Journal of Experimental Psychology: Learning, Memory, and Cognition, 2004
- Elimination and Inclusion Procedures in JudgmentJournal of Behavioral Decision Making, 1997
- Effects of task format on probabilistic forecasting of stock pricesInternational Journal of Forecasting, 1996
- Graininess of judgment under uncertainty: An accuracy-informativeness trade-off.Journal of Experimental Psychology: General, 1995
- Calibration of probabilities: The state of the art to 1980Published by Cambridge University Press (CUP) ,1982
- Do those who know more also know more about how much they know?Organizational Behavior and Human Performance, 1977
- Overconfidence in case-study judgments.Journal of Consulting Psychology, 1965