Experimentation
A/B Test Significance Calculator
Check if your A/B test results are statistically significant and estimate the sample size needed for reliable conclusions.
Result
Not significant yet. Continue the test or increase traffic to reach a reliable conclusion.
Visitors per variant for 80% power to detect the observed difference at 95% confidence. You have 5,000 (31% of it).
Conversion rate comparison
Control (A) vs variant (B) conversion rates.
How to use it
- Enter visitors and conversions for control A and variant B, then choose a confidence level. Use 95% for most product and marketing decisions and 99% when the change affects revenue, compliance, or a large user population.
- Read both conversion rates, relative lift, z-score, p-value, conclusion, required sample size, and the power message. A Borderline result means the data is close enough that peeking early could easily push you into a false decision.
- Interpret significance and effect size together. A result can be statistically significant but too small to matter commercially, while a large-looking lift with a Not Significant label usually means you need more traffic before shipping anything.
- Use the required sample size to decide whether to continue, stop, or redesign the experiment. Predefine the minimum lift worth shipping so a tiny 0.1-0.2 point improvement does not consume engineering effort with no meaningful business return.
- Re-run only after full business cycles or materially more traffic arrives. Track win rate and realized post-launch lift by experiment type so your testing program learns which kinds of hypotheses actually produce durable gains.
Questions people usually ask
What sample size do I need for a valid A/B test?
It depends on your baseline conversion rate and minimum detectable effect. To detect a 20% relative improvement on a 5% baseline (from 5% to 6%) at 95% confidence and 80% statistical power, this tool's two-tailed formula gives approximately 8,200 visitors per variant. Smaller effects require dramatically larger samples — detecting a 10% relative improvement (5% to 5.5%) needs roughly 31,000 per variant.
What is the difference between statistical significance and practical significance?
A test can be statistically significant (very unlikely to be due to chance) but practically insignificant (effect too small to matter). A 0.1% conversion rate improvement may be p<0.01 with 500,000 visitors but generate only $200/month in additional revenue. Always evaluate effect size alongside p-value — significance without magnitude is misleading.
How long should I run an A/B test?
Run for at least 1-2 full business cycles (usually 2-4 weeks minimum) regardless of when significance is reached. Stopping the moment significance appears (peeking) inflates the false-positive rate well above the nominal 5%, because a fixed-sample z-test assumes you look once. Pre-specify the sample size before launching.
Related calculators
- Marketing Churn & Retention Calculator
Estimate recovered customers and revenue lift from retention improvements.
- Marketing CAC Calculator
Calculate customer acquisition cost, payback period, and LTV:CAC efficiency.
- Marketing Ad Spend / ROAS Calculator
Calculate actual ROAS, break-even ROAS, profit after ad spend, target CPA, and required conversion rate for advertising campaigns.
- Marketing Email Marketing ROI Calculator
Calculate projected revenue, ROI, cost per acquisition, and break-even conversion rate for email marketing campaigns.
Background on the A/B Test Significance Calculator
- Learn What Is Minimum Viable Experiment? Simply Explained
A minimum viable experiment tests one hypothesis with the smallest intervention that can answer it. The definition, how to design one and an example.
Articles
- 9 min read How to Run an A/B Test: Sample Size, Duration, Results
How to run an A/B test: one hypothesis and metric, sample size fixed before launch (24,198 per arm for a 3% to 3.45% lift), a full week and no peeking.
- 7 min read How to Optimize Your Conversion Rate
Optimize conversion rate by prioritising experiments with ICE scores, powering tests to detect realistic lifts, and avoiding the peeking trap.
More in Marketing and acquisition
Know whether your marketing spend is building value or burning cash.
- CAC Calculator
- Churn & Retention Calculator
- Ad Spend / ROAS Calculator
- Email Marketing ROI Calculator
Use it from code
The same calculation as a JavaScript module you can import. It runs where you import it, with no request, key or rate limit.
import { compute } from "https://aibizhub.io/engines/ab-test-significance-calculator.js";Input and output contract, TypeScript types and the guide for agents.