EXPERIMENTATION · STATISTICS
A/B Test Significance Calculator
Check if your A/B test results are statistically significant and estimate the sample size needed for reliable conclusions.
Result
Not significant yet. Continue the test or increase traffic to reach a reliable conclusion.
Estimated visitors per variant for 80% power at the selected confidence level.
Conversion rate comparison
Control (A) vs variant (B) conversion rates.
How to use it
- Enter visitors and conversions for control A and variant B, then choose a confidence level. Use 95% for most product and marketing decisions and 99% when the change affects revenue, compliance, or a large user population.
- Read both conversion rates, relative lift, z-score, p-value, conclusion, required sample size, and the power message. A Borderline result means the data is close enough that peeking early could easily push you into a false decision.
- Interpret significance and effect size together. A result can be statistically significant but too small to matter commercially, while a large-looking lift with a Not Significant label usually means you need more traffic before shipping anything.
- Use the required sample size to decide whether to continue, stop, or redesign the experiment. Predefine the minimum lift worth shipping so a tiny 0.1-0.2 point improvement does not consume engineering effort with no meaningful business return.
- Re-run only after full business cycles or materially more traffic arrives. Track win rate and realized post-launch lift by experiment type so your testing program learns which kinds of hypotheses actually produce durable gains.
Questions people usually ask
What sample size do I need for a valid A/B test?
It depends on your baseline conversion rate and minimum detectable effect. To detect a 20% relative improvement on a 5% baseline (from 5% to 6%) at 95% confidence and 80% statistical power, you need approximately 4,800 visitors per variant. Smaller effects require dramatically larger samples — detecting a 10% relative improvement requires roughly 19,000 per variant.
What is the difference between statistical significance and practical significance?
A test can be statistically significant (very unlikely to be due to chance) but practically insignificant (effect too small to matter). A 0.1% conversion rate improvement may be p<0.01 with 500,000 visitors but generate only $200/month in additional revenue. Always evaluate effect size alongside p-value — significance without magnitude is misleading.
How long should I run an A/B test?
Run for at least 1-2 full business cycles (usually 2-4 weeks minimum) regardless of when significance is reached. Stopping early when significance appears inflates false positive rates significantly — the peaking-at-significance problem can produce 30-50% of results that fail to replicate. Pre-specify sample size before launching.
Related Resources
Learn the decision before you act
Every link here is tied directly to A/B Test Significance Calculator. Use the explanation, formula, examples, and benchmarks to pressure-test the calculator output from first principles.
How To Use
4 STEPSHow to Use Landing Page Conversion Calculator
Project landing page revenue and ROI from visitor traffic, conversion rate, and average order value. Optimize page performance with clear unit economics.
ReadHow To Use
5 STEPSHow to Use A/B Test Significance Calculator
Validate your A/B test results to make data-driven decisions. Learn how to use this calculator to determine if observed differences in conversion rates.
ReadHow To Use
5 STEPSHow to Use CAC Calculator
CAC = fully loaded sales + marketing spend ÷ new customers. Check it against LTV:CAC to see whether to scale acquisition or fix unit economics first.
ReadHow To Use
5 STEPSHow to Use Churn & Retention Calculator
Master customer loyalty and growth with our Churn & Retention Calculator. Understand your customer base, identify loss trends, and strategize.
ReadFormula
3 VARIABLESPrice Elasticity Formula
Uncover customer demand sensitivity to price changes with the Price Elasticity Formula. Essential for optimizing pricing strategies through data-driven.
ReadExamples
4 EXAMPLESAb Test Significance Examples
Master A/B test significance with practical examples across diverse industries. Learn to interpret results beyond p-values for real business impact.
ReadRelated deep dive
All articles →Read further
Long-form context behind the calculator output.
- Article·8 min
How to Run A/B Tests That Actually Work
Run A/B tests that survive replication: pick one metric, power the test, stop peeking, and read results with the practical-significance check most teams skip.
Read - Article·7 min
How to Optimize Your Conversion Rate
Optimize conversion rate by prioritising experiments with ICE scores, powering tests to detect realistic lifts, and avoiding the peeking trap.
Read - Article·9 min
Landing Page Conversion at 2%: The Solo Founder Floor
Landing page conversion at 1.9% on 12,000 monthly visitors, $84 AOV, $640 cost. The page pays only when AOV moves — not when conversion does.
Read
Continue With Related Tools
See the revenue impact of reducing churn — even a small improvement compounds into significant retained revenue over time.
Open →Know what each new customer really costs — CAC, payback period, and LTV:CAC health from your actual spend and revenue numbers.
Open →Decide whether to scale or cut ad spend — actual ROAS, break-even ROAS, profit, and target CPA for your campaigns.
Open →Decide if your email campaigns are worth the spend — projected revenue, ROI, CPA, and break-even conversion rate.
Open →More in Marketing & Acquisition
Know whether your marketing spend is building value or burning cash.
Decision Workflows
Step-by-step guides that use this tool.