Worked examples, in Experimentation
A/B Test Significance Examples & P-Values
You ran the test, one variant won — but was the lift real or just noise? Calling a winner too early on thin data wastes budget and poisons future decisions. These worked examples show how to read p-values and confidence intervals so you ship the variant that actually converted better.
4 examples. Published Mar 12, 2026. Updated Mar 27, 2026. By AI Biz Hub.
Worked examples
See the inputs and outcome together
Each scenario keeps the starting point, the outcome, and the actual lesson in one place so the page reads like a decision notebook, not a data dump.
- 1
Baseline case
Compare a control and variant that each drew 5,000 visitors, with B converting 285 against A's 250.
Rate A is 5.0% and rate B is 5.7%, a 14% relative lift. But the p-value is 0.12 and the verdict is Not Significant: you would need about 16,224 visitors per arm to trust it.
Visitors A
5,000
Conversions A
250
Visitors B
5,000
Conversions B
285
A 14% lift looks promising yet fails the test at this sample size. Resist shipping on the headline number; the math says keep the test running until traffic catches up.
- 2
Stronger variant
Keep the 5,000 visitors per arm but let B convert 320, widening its edge over the control.
Rate B rises to 6.4% versus 5.0% for A, a 28% relative lift. The p-value drops to 0.0025 and the verdict flips to Significant.
Visitors A
5,000
Conversions A
250
Visitors B
5,000
Conversions B
320
Doubling the effect size from 14% to 28% pushed the same traffic past the bar. Bigger wins reach significance faster, which is why a clearly better variant settles quickly.
- 3
Same lift, more traffic
Reproduce the baseline 14% lift but quadruple the sample: 20,000 visitors per arm, 1,000 against 1,140 conversions.
Rates are still 5.0% and 5.7% with the same 14% lift, but the p-value now falls to 0.0019 and the verdict is Significant.
Visitors A
20,000
Conversions A
1,000
Visitors B
20,000
Conversions B
1,140
The effect size is identical to the baseline, yet four times the traffic turned a coin-flip into a confident win. Sample size, not just the lift, decides whether a result holds.
- 4
Marginal variant
Hold 5,000 visitors per arm and shrink B's edge to 260 conversions, a barely-there improvement.
Rate B is 5.2% against 5.0%, a 4% lift with a p-value of 0.65 and a Not Significant verdict. Detecting it reliably would take roughly 189,992 visitors per arm.
Visitors A
5,000
Conversions A
250
Visitors B
5,000
Conversions B
260
A 4% lift is statistical noise at any realistic traffic level. The required sample of nearly 190,000 per arm is a signal to stop the test and chase a bolder change instead.
Patterns
Run the numbers next
- Run the Numbers Net Promoter Score (NPS) Calculator
Calculate NPS from promoter, passive, and detractor counts with benchmark context and action guidance.
- Run the Numbers Churn & Retention Calculator
Estimate recovered customers and revenue lift from retention improvements.
Questions people ask next
What does statistical significance mean in an A/B test?
A/B test significance tells you whether the observed difference between two variants is likely caused by the change you made or is just random chance. It is what lets you call a real winner instead of shipping noise, and it is read from the p-value and confidence interval, not from the headline lift.
Why can a 14% lift still be not significant?
In the baseline example, control and variant each drew 5,000 visitors and B converted at 5.7% versus 5.0% for A, a 14% relative lift, yet the p-value was 0.12 and the verdict was Not Significant. At that sample size you would need about 16,224 visitors per arm to trust the result, so a promising headline number can still fail the test.
Does more traffic make an A/B test result significant?
It can. The 'same lift, more traffic' example reproduces the same 14% lift (5.0% vs 5.7%) but quadruples the sample to 20,000 visitors per arm, and the p-value falls to 0.0019 with a Significant verdict. The effect size is identical to the not-significant baseline, so sample size, not just the lift, decides whether a result holds.
How big a sample do I need to detect a small lift?
Very large. The marginal example holds 5,000 visitors per arm with B at 5.2% versus 5.0% (a 4% lift), giving a p-value of 0.65 and a Not Significant verdict; detecting it reliably would take roughly 189,992 visitors per arm. A 4% lift is effectively statistical noise at realistic traffic, a signal to chase a bolder change instead.
Sources
Keep the topic connected
- Learn What Is Churn Rate? Simply Explained
Understand Churn Rate, a vital SaaS metric, with a clear definition, formula, real-world examples, and strategies to improve customer retention and revenue.