A/B Test Significance: worked example
| Visitors, version A | 5,000 |
|---|---|
| Conversions, version A | 250 |
| Visitors, version B | 5,000 |
| Conversions, version B | 300 |
| Relative uplift of B over A | 20.0% |
| Z-score | 2.2 |
| Two-tailed p-value | 0.028 |
A/B Test Significance: formula
U.S. SBA: manage your finances · NIST: Critical values and p-values
Common error: Reading a p-value of 0.028 as a 97 percent chance that B is better. It is the chance of a gap this large if there were no real difference, which is a different statement.
Each A/B arm has its own visitor count and conversion count. Concurrent random assignment is an assumption of the comparison. Both groups should run for the same period, ideally a full week or more to capture weekday effects, and you should decide the sample size before starting rather than stopping when the result looks good.
Count each visitor once and exclude bots.
A/B Test Significance: changing Visitors, version A
| Visitors, version A | Relative uplift of B over A | Z-score | Two-tailed p-value |
|---|---|---|---|
| 4,000 | -4.0% | -0.492 | 0.623 |
| 4,500 | 8.0% | 0.926 | 0.354 |
| 5,000 (example) | 20.0% | 2.2 | 0.028 |
| 5,500 | 32.0% | 3.3 | 0.00083 |
| 6,000 | 44.0% | 4.4 | 0.00001 |
A/B Test Significance: compare input ranges
| Input | Tested from | To | Relative uplift of B over A: low input | Relative uplift of B over A: high input | Swing |
|---|---|---|---|---|---|
| Conversions, version B | 270 | 330 | 8.0% | 32.0% | 24.0% (120%) |
| Visitors, version A | 4,500 | 5,500 | 8.0% | 32.0% | 24.0% (120%) |
| Visitors, version B | 4,500 | 5,500 | 33.3% | 9.1% | 24.2% (121%) |
| Conversions, version A | 225 | 275 | 33.3% | 9.1% | 24.2% (121%) |
A/B Test Significance: individual input comparisons
| Input | Value | Relative uplift of B over A | Z-score | Two-tailed p-value |
|---|---|---|---|---|
| Visitors, version A | 4,000 | -4.0% | -0.492 | 0.623 |
| 4,500 | 8.0% | 0.926 | 0.354 | |
| 5,000 | 20.0% | 2.2 | 0.028 | |
| 5,500 | 32.0% | 3.3 | 0.00083 | |
| 6,000 | 44.0% | 4.4 | 0.00001 | |
| Conversions, version A | 200 | 50.0% | 4.6 | 0 |
| 225 | 33.3% | 3.4 | 0.00077 | |
| 250 | 20.0% | 2.2 | 0.028 | |
| 275 | 9.1% | 1.1 | 0.283 | |
| 300 | 0.000% | 0 | 1 | |
| Visitors, version B | 4,000 | 50.0% | 4.9 | 0 |
| 4,500 | 33.3% | 3.5 | 0.00051 | |
| 5,000 | 20.0% | 2.2 | 0.028 | |
| 5,500 | 9.1% | 1 | 0.296 | |
| 6,000 | 0.000% | 0 | 1 | |
| Conversions, version B | 240 | -4.0% | -0.463 | 0.643 |
| 270 | 8.0% | 0.901 | 0.368 | |
| 300 | 20.0% | 2.2 | 0.028 | |
| 330 | 32.0% | 3.4 | 0.00062 | |
| 360 | 44.0% | 4.6 | 0 |
A/B Test Significance: combined input changes
| Visitors, version A / Conversions, version A | 200 | 250 | 300 |
|---|---|---|---|
| 4,000 | 20.0% | -4.0% | -20.0% |
| 4,500 | 35.0% | 8.0% | -10.0% |
| 5,000 | 50.0% | 20.0% | 0.000% |
| 5,500 | 65.0% | 32.0% | 10.0% |
| 6,000 | 80.0% | 44.0% | 20.0% |
A/B Test Significance: which assumptions move the result?
A/B Test Significance: values and definitions
| Quantity and definition | Value used |
|---|---|
| Visitors, version A | 5,000 |
| Conversions, version A | 250 |
| Visitors, version B | 5,000 |
| Conversions, version B | 300 |
| Relative uplift of B over A | 20.0% |
| Z-score | 2.2 |
| Two-tailed p-value | 0.028 |
Checking results repeatedly and stopping early raises the false-positive rate, as does running many tests and picking winners. A significant result on a tiny lift may not be worth the change. The test assumes independent visitors and a fixed sample size.
It says nothing about why the difference occurred or whether it will last.
Business Finance calculators: Overhead Rate Calculator, Gross Profit and Margin Calculator, Markup Calculator, Business Break-Even Calculator, Profit Margin Calculator, Burn Rate & Runway Calculator.
An A/B test compares two conversion rates, and the question is whether the gap is bigger than chance would produce. With 250 conversions from 5,000 visitors against 300 from 5,000, version B is up 20 percent, the z-score is about 2.19, and the p-value is about 0.028. That is below 0.05, the usual bar, so the gap is unlikely to be luck alone, though it is not proof.
Repeated early stopping changes the interpretation of a fixed-sample significance calculation. Stopping the moment a result looks good raises the false positive rate well above the stated p-value.
Calculate baseline rates with the conversion rate calculator, and measure the business impact with the ecommerce profit and ROAS pages. The probability and standard deviation calculators support the statistics. Plan sample sizes in advance, since small differences need thousands of visitors.
What does a p-value of 0.03 mean?
A two-sided p-value of 0.03 means that, under the null model and its assumptions, results at least as extreme as the observed statistic have probability 3%. It is not the probability that the null hypothesis is true.
Can I stop the test as soon as it is significant?
No. Stopping early inflates false positives. Set the sample size or duration in advance and wait until it is reached.
Is statistical significance the same as importance?
No. A tiny lift can be significant with lots of traffic but not worth acting on. Judge the size of the effect and its value too.