Learn what a p-value does and does not say, how to run a fair test, and why a positive result from a good test can still be wrong. Flags the 4 mistakes that most often produce a plausible-but-wrong answer.
Key takeaways
- The z-score measures how many standard errors apart two rates are, and the p-value converts it to a probability.
- A p-value is the chance of the data under no effect, not the chance the effect is real.
- Fix the sample size or stopping rule before the test, and do not stop at the first significant result.
- Use two tails unless you chose a direction before seeing the data.
- When a condition is rare, most positive results come from the healthy group, even for an accurate test.
What an A/B test asks
An A/B test splits visitors between two versions and compares how often each converts. The question is whether the difference in conversion rates is bigger than random variation would produce. Even two identical pages show slightly different rates because visitors differ. The test measures the size of the gap against that ordinary noise.
The z-score turns a gap into a distance
For two conversion rates, the z-score divides the difference between them by its standard error. It counts how many standard errors apart the two rates are. A larger absolute value means a gap that is harder to explain by chance. With 5 percent against 6 percent on 5,000 visitors each, the z-score is about 2.19.
What a p-value is
The p-value is the probability of seeing a gap at least this large if there were no real difference between the versions. A p-value of 0.028 means that a gap this big would appear about 2.8 percent of the time by chance alone. It is a statement about the data under the assumption of no effect, and it is not the probability that the hypothesis is true.
What a p-value is not
A p-value is not the chance that version B is better, not the size of the effect and not a measure of importance. A tiny difference can be highly significant with a huge sample, and a large difference can be non-significant with a small one. Always read the effect size beside the p-value, and ask whether the gap is worth acting on.
One tail or two
A two-tailed test counts a gap in either direction, and a one-tailed test counts only one. Use two tails unless you committed to a direction before seeing the data. Choosing the tail after the result inflates false positives, because it halves the p-value in a way the design did not earn.
Fix the sample size before you start
Stopping a test the moment the result looks significant raises the false positive rate well above the stated level. Decide the sample size, or a fixed stopping rule, in advance and stick to it. A test that runs until it finds significance will find it eventually, even with no real difference.
Run for whole cycles
Behavior changes by day of the week and by season. A test that runs only on weekdays or across a holiday can mislead. Run for at least one full business cycle, often one or more whole weeks, so each version sees the same mix of visitors.
Bayes' theorem updates a belief
Bayes' theorem combines a starting probability with new evidence. It says the chance of a condition after a positive test depends on how common the condition is, how often the test catches it and how often it gives false alarms. It is the tool for turning a test result into an honest probability.
Why a good test can still be wrong
Take a condition that affects 1 percent of people and a test that is 90 percent sensitive with a 5 percent false positive rate. A positive result means only about a 15 percent chance of the condition. Most positives come from the large healthy group, since 5 percent of a big group outnumbers 90 percent of a small one. Rarity of the condition is what drives this.
Using these tools with care
Statistics describe uncertainty, and they do not remove it. Report the effect size, the sample and the test used. Treat a single significant result as a lead to confirm, and a non-significant one as an unknown, not proof of no effect. When a decision is costly, gather more data or repeat the test.
Worked with real numbers
What this looks like in the a/b test significance calculator
The guidance above is easier to judge against figures. Using the a/b test significance calculator worked example, moving visitors, version a from 4,000 to 6,000 changes the relative uplift of b over a from -4.0% to 44.0%.
| Visitors, version A | Relative uplift of B over A | Z-score | Two-tailed p-value |
|---|---|---|---|
| 4,000 | -4.0% | -0.492 | 0.623 |
| 4,500 | 8.0% | 0.926 | 0.354 |
| 5,000 | 20.0% | 2.2 | 0.028 |
| 5,500 | 32.0% | 3.3 | 0.00083 |
| 6,000 | 44.0% | 4.4 | 0.00001 |
Open the A/B Test Significance Calculator to use your own numbers →
The inputs behind those figures
| Input | Value | Definition |
|---|---|---|
| Visitors, version A | 5,000 | Enter the visitors, version a used in this calculation. |
| Conversions, version A | 250 | Enter the conversions, version a used in this calculation. |
| Visitors, version B | 5,000 | Enter the visitors, version b used in this calculation. |
| Conversions, version B | 300 | Enter the conversions, version b used in this calculation. |
What goes wrong
Common mistakes
Each of these produces an answer that looks reasonable, which is why they survive review. To catch them in a/b tests, p-values and bayes' theorem, rerun the a/b test significance calculator with a different assumption and check whether the result moves in the direction the guidance predicts, since an error that survives that test is usually in one of the inputs and not in the arithmetic.
| The mistake | Why it misleads | Do this instead |
|---|---|---|
| Stopping when the result first looks significant | Peeking and stopping early makes a false positive far more likely than the stated p-value suggests. | Set the sample size or stopping rule in advance and keep to it. |
| Reading p = 0.03 as a 97 percent chance the effect is real | The p-value is about the data under no effect, and it does not give the probability of a hypothesis. | Say what it is: the chance of a gap this large if nothing were going on. |
| Ignoring the effect size | A very small gap can be significant in a huge sample and not worth acting on. | Read the relative uplift and its practical value beside the p-value. |
| Forgetting the base rate | A positive test on a rare condition is often a false alarm, even when the test is accurate. | Apply Bayes' theorem with the prevalence for the group being tested. |
People also ask
Frequently asked questions
What is a p-value?
The probability of seeing a result at least as extreme as yours if there were no real effect. A small p-value suggests the result is unlikely to be chance alone.
What p-value counts as significant?
0.05 is a common threshold, but it is a convention. The right level depends on the cost of a wrong decision, and a result near the line is weak evidence.
How do I compare two conversion rates?
Compute the pooled z-test: divide the difference in rates by its standard error and convert the z-score to a p-value. The A/B test significance calculator does it.
What is Bayes' theorem?
A formula that updates the probability of a condition given new evidence, using the base rate, the test's sensitivity and its false positive rate.
Why is a positive result often wrong when the condition is rare?
Because most people tested do not have the condition, so even a small false positive rate produces more false alarms than true positives.
Should I use a one-tailed or two-tailed test?
Two-tailed unless you decided on a direction before you saw the data. Picking the tail afterward inflates false positives.
How long should I run an A/B test?
Long enough to reach a sample size fixed in advance and to cover at least one full business cycle, so weekday and weekend behavior are both included.
What sample size do I need for an A/B test?
It depends on the baseline conversion rate, the smallest lift you care about and the confidence you want. Smaller lifts and lower baseline rates need many more visitors.
Can I run several tests at once?
Yes if they touch different parts of the experience and different visitors, but many tests at once raise the chance that one looks significant by luck. Adjust the threshold or confirm winners.
Which input moves the relative uplift of b over a most in the a/b test significance calculator?
Ranked by how far each moves the relative uplift of b over a across the range tested: conversions, version b (24.0%, 120%), visitors, version a (24.0%, 120%), visitors, version b (24.2%, 121%) and conversions, version a (24.2%, 121%).
How much does visitors, version a matter in the a/b test significance calculator?
The worked example uses 5,000. With the other inputs left at the worked example, moving visitors, version a from 4,500 to 5,500 takes the relative uplift of b over a from 8.0% to 32.0%, a swing of 120% of the worked-example figure.