Methodology & Meta-science

The fallacy of placing confidence in confidence intervals

Interval estimates -- estimates of parameters that include an allowance for sampling uncertainty -- have long been touted as a key component of statistical analyses. There are several kinds of interval estimates, but the most popular are confidence intervals (CIs): intervals that contain the true parameter value in some known proportion of repeated samples, on average. The width of confidence intervals is thought to index the precision of an estimate; the parameter values contained within a CI are thought to be more plausible than those outside the interval; and the confidence coefficient of the interval (e.g. 95%) is thought to index the plausibility that the true parameter is included in the interval. CIs do not necessarily have any of these properties; given that confidence intervals do not necessarily have these desirable properties, their use may lead to unjustified or arbitrary inferences if researchers rely on them. For this reason, we caution against the use of confidence interval theory to justify interval estimates.