Confidence interval

Before you rely on it

What to check

Note the confidence level and the sample size behind it.

The common error

Where people go wrong with confidence interval

Reporting two figures as different when their intervals overlap. That is not a finding, it is noise given a headline.

Worked through

Confidence interval in plain numbers

A survey of 400 people finds that 52% favor a proposal. With that sample size the standard error is about the square root of 0.52 × 0.48 ÷ 400, or 2.5 points; a 95% confidence interval runs roughly two standard errors either side: 52% ± 4.9, so from about 47% to 57%; the range includes 50%. So the survey cannot say whether the proposal has majority support. Survey 1,600 people and the same 52% would carry a margin near 2.4 points, and the interval of 49.6% to 54.4% would lean toward a majority. Quadruple the sample, halve the width.

Easily confused

Confidence interval and its neighbours

A confidence interval is not a probability that the true value lies inside this particular range; the 95% describes the method: if you repeated the study many times and built an interval each time, about 95% of those intervals would contain the true value. It is also different from a prediction interval, which covers where a future single observation may fall, and from a range of the data itself; the margin of error is half the width of the interval. Standard deviation describes the spread of the data. The interval describes uncertainty about an estimate.

In real decisions

Where confidence interval turns up

Polls, clinical trials, quality control and A/B tests all report confidence intervals; a drug study might say a treatment cut risk by 20% with an interval from 5% to 33%; a manufacturer checks that the mean part diameter sits inside a tolerance. A marketer compares two conversion rates and asks whether their intervals overlap. The interval shows both the size of an effect and how sure the study is about it, and that beats a single number every time.

Know the edges

Where confidence interval stops being reliable

Intervals assume a random sample and an appropriate model. Biased samples, low response rates and badly worded questions are not fixed by a wider interval. Small samples give wide ranges. Multiple comparisons make some intervals miss by chance. The 95% level is a convention, not a law; a narrow interval around a biased estimate is still wrong, and treat the interval as a measure of sampling noise only, not of every source of error.

Quick answers

What people ask about confidence interval

What does 95% confidence mean? If you repeated the study many times and built an interval each time, about 95 of every 100 intervals would contain the true value. It is a statement about the method. It does not give a 95% chance that this one interval holds the truth.

How can I get a narrower interval? Use a larger sample, or accept a lower confidence level. Quadrupling the sample roughly halves the width. Lower variability in the data helps too.