A pooled two-proportion comparison
Assume independent randomized groups: 250 conversions among 5,000 visitors versus 300 among 5,000. The pooled rate is 0.055. Standard error under equal rates is √(0.055 × 0.945 × (1/5000 + 1/5000)); the observed one-percentage-point gap gives z ≈ 2.193.
Repeated peeking and multiple comparisons require a different error-control plan.
A p-value conditions on the null model
For a pre-specified two-sided pooled normal test in that example, p is approximately 0.0283. It is the null-model probability of a test statistic at least as extreme as observed, given the test assumptions. It is not the probability that no effect exists or that the next experiment will repeat the result.
What a p-value is not
A p-value is not the chance that version B is better, not the size of the effect and not a measure of importance. A tiny difference can be highly significant with a huge sample, and a large difference can be non-significant with a small one. Always read the effect size beside the p-value, and ask whether the gap is worth acting on.
A posterior needs a prior and a likelihood
In a hypothetical classification example, assume prevalence 1%, sensitivity 90% and false-positive rate 5%. Among 10,000 cases, expected true positives are 90 and false positives 495. The posterior fraction is 90/(90 + 495) = 15.3846%.
This conditions on a positive result and differs from sensitivity, which conditions on having the condition. It is an assumed probability model, not a clinical prediction about an individual.
Repeated looks change the testing procedure
Stopping a test the moment the result looks significant raises the false positive rate well above the stated level. Decide the sample size, or a fixed stopping rule, in advance and stick to it. A test that runs until it finds significance will find it eventually, even with no real difference.
Run for whole cycles
Behavior changes by day of the week and by season. A test that runs only on weekdays or across a holiday can mislead. Run for at least one full business cycle, often one or more whole weeks, so each version sees the same mix of visitors.
Bayes' theorem updates a belief
Bayes' theorem combines a starting probability with new evidence. It says the chance of a condition after a positive test depends on how common the condition is, how often the test catches it and how often it gives false alarms. It is the tool for turning a test result into a probability.
Why a good test can still be wrong
Take a condition that affects 1 percent of people and a test that is 90 percent sensitive with a 5 percent false positive rate. A positive result means only about a 15 percent chance of the condition. Most positives come from the large healthy group, since 5 percent of a big group outnumbers 90 percent of a small one.
Rarity of the condition is what drives this.
Using these tools with care
Statistics describe uncertainty, and they do not remove it. Report the effect size, the sample and the test used. One significant result can warrant replication.
A nonsignificant result can also reflect insufficient statistical power; it does not establish zero effect. When a decision is costly, gather more data or repeat the test.
Assumed inputs and calculated outputs
A/B Test Significance Calculator: input sensitivity
| Visitors, version A | Relative uplift of B over A | Z-score | Two-tailed p-value |
|---|---|---|---|
| 4,000 | -4.0% | -0.492 | 0.623 |
| 4,500 | 8.0% | 0.926 | 0.354 |
| 5,000 | 20.0% | 2.2 | 0.028 |
| 5,500 | 32.0% | 3.3 | 0.00083 |
| 6,000 | 44.0% | 4.4 | 0.00001 |
Open the A/B Test Significance Calculator to use your own numbers →
What goes wrong
A/B tests, p-values and Bayes' theorem: common errors
| The mistake | Why it misleads | Do this instead |
|---|---|---|
| Stopping when the result first looks significant | Peeking and stopping early makes a false positive far more likely than the stated p-value suggests. | Set the sample size or stopping rule in advance and keep to it. |
| Reading p = 0.03 as a 97 percent chance the effect is real | The p-value is about the data under no effect, and it does not give the probability of a hypothesis. | Say what it is: the chance of a gap this large if nothing were going on. |
| Ignoring the effect size | A very small gap can be significant in a huge sample and not worth acting on. | Read the relative uplift and its practical value beside the p-value. |
| Forgetting the base rate | A positive test on a rare condition is often a false alarm, even when the test is accurate. | Apply Bayes' theorem with the prevalence for the group being tested. |
A/B tests, p-values and Bayes' theorem: worked calculation
Using the A/B Test Significance Calculator worked example, moving Visitors, version A from 4,000 to 6,000 changes the Relative uplift of B over A from -4.0% to 44.0%.
| Input | Value | Definition |
|---|---|---|
| Visitors, version A | 5,000 | Entered example assumption. |
| Conversions, version A | 250 | Entered example assumption. |
| Visitors, version B | 5,000 | Entered example assumption. |
| Conversions, version B | 300 | Entered example assumption. |
| Relative uplift of B over A | 20.0% |
|---|---|
| Z-score | 2.2 |
| Two-tailed p-value | 0.028 |
| Visitors, version A | Relative uplift of B over A | Z-score | Two-tailed p-value |
|---|---|---|---|
| 4,000 | -4.0% | -0.492 | 0.623 |
| 4,500 | 8.0% | 0.926 | 0.354 |
| 5,000 | 20.0% | 2.2 | 0.028 |
| 5,500 | 32.0% | 3.3 | 0.00083 |
| 6,000 | 44.0% | 4.4 | 0.00001 |
Which input moves the relative uplift of b over a most in the a/b test significance calculator?
Ranked by how far each moves the relative uplift of b over a across the range tested: conversions, version b (24.0%, 120%), visitors, version a (24.0%, 120%), visitors, version b (24.2%, 121%) and conversions, version a (24.2%, 121%).
How much does conversions, version a matter in the a/b test significance calculator?
The worked example uses 250. Holding every other input at its worked-example value, moving conversions, version a from 225 to 275 takes the relative uplift of b over a from 33.3% to 9.1%, a swing of 121% of the worked-example figure.
A/B tests, p-values and Bayes' theorem: points to carry forward
- The z-score measures how many standard errors apart two rates are, and the p-value converts it to a probability.
- A p-value measures the probability of data at least this extreme under the stated null model. It does not assign a probability that the effect is real.
- For twenty independent tests whose null hypotheses are all true, each with a 5% false-positive probability, the chance of at least one false positive is 1 − 0.95²⁰, about 64.15%. This calculation concerns independent tests, not an exact estimate for sequential looks at one experiment.
- Use two tails unless you chose a direction before seeing the data.
- When a condition is rare, most positive results come from the healthy group, even for an accurate test.
What is a p-value?
The probability of seeing a result at least as extreme as yours if there were no real effect. A small p-value suggests the result is unlikely to be chance alone.
What p-value counts as significant?
0.05 is a common threshold, but it is a convention. The right level depends on the cost of a wrong decision, and a result near the line is weak evidence.
How do I compare two conversion rates?
Compute the pooled z-test: divide the difference in rates by its standard error and convert the z-score to a p-value. The A/B test significance calculator does it.
What is Bayes' theorem?
A formula that updates the probability of a condition given new evidence, using the base rate, the test's sensitivity and its false positive rate.
Why is a positive result often wrong when the condition is rare?
Because most people tested do not have the condition, so even a small false positive rate produces more false alarms than true positives.
Should I use a one-tailed or two-tailed test?
Two-tailed unless you decided on a direction before you saw the data. Picking the tail afterward inflates false positives.
How long should I run an A/B test?
Long enough to reach a sample size fixed in advance and to cover at least one full business cycle, so weekday and weekend behavior are both included.
What sample size do I need for an A/B test?
It depends on the baseline conversion rate, the smallest lift you care about and the confidence you want. Smaller lifts and lower baseline rates need many more visitors.
Can I run several tests at once?
Yes if they touch different parts of the experience and different visitors, but many tests at once raise the chance that one looks significant by luck. Adjust the threshold or confirm winners.