Experimentation
A/B Test Significance Calculator
Check if your A/B test results are statistically significant and estimate the sample size needed for reliable conclusions.
Result
Not significant yet. Continue the test or increase traffic to reach a reliable conclusion.
Visitors per variant for 80% power to detect the observed difference at 95% confidence. You have 5,000 (31% of it).
Conversion rate comparison
Control (A) vs variant (B) conversion rates.
How to use it
- Enter visitors and conversions for control A and variant B, then choose a confidence level. Use 95% for most product and marketing decisions and 99% when the change affects revenue, compliance, or a large user population.
- Read both conversion rates, relative lift, z-score, p-value, conclusion, required sample size, and the power message. A Borderline result means the data is close enough that peeking early could easily push you into a false decision.
- Interpret significance and effect size together. A result can be statistically significant but too small to matter commercially, while a large-looking lift with a Not Significant label usually means you need more traffic before shipping anything.
- Use the required sample size to decide whether to continue, stop, or redesign the experiment. Predefine the minimum lift worth shipping so a tiny 0.1-0.2 point improvement does not consume engineering effort with no meaningful business return.
- Re-run only after full business cycles or materially more traffic arrives. Track win rate and realized post-launch lift by experiment type so your testing program learns which kinds of hypotheses actually produce durable gains.
Questions people usually ask
What sample size do I need for a valid A/B test?
It depends on your baseline conversion rate and minimum detectable effect. To detect a 20% relative improvement on a 5% baseline (from 5% to 6%) at 95% confidence and 80% statistical power, this tool's two-tailed formula gives approximately 8,200 visitors per variant. Smaller effects require dramatically larger samples — detecting a 10% relative improvement (5% to 5.5%) needs roughly 31,000 per variant.
What is the difference between statistical significance and practical significance?
A test can be statistically significant (very unlikely to be due to chance) but practically insignificant (effect too small to matter). A 0.1% conversion rate improvement may be p<0.01 with 500,000 visitors but generate only $200/month in additional revenue. Always evaluate effect size alongside p-value — significance without magnitude is misleading.
How long should I run an A/B test?
Run for at least 1-2 full business cycles (usually 2-4 weeks minimum) regardless of when significance is reached. Stopping the moment significance appears (peeking) inflates the false-positive rate well above the nominal 5%, because a fixed-sample z-test assumes you look once. Pre-specify the sample size before launching.
Related calculators
- Marketing Churn & Retention Calculator
Estimate recovered customers and revenue lift from retention improvements.
- Marketing CAC Calculator
Calculate customer acquisition cost, payback period, and LTV:CAC efficiency.
- Marketing Ad Spend / ROAS Calculator
Calculate actual ROAS, break-even ROAS, profit after ad spend, target CPA, and required conversion rate for advertising campaigns.
- Marketing Email Marketing ROI Calculator
Calculate projected revenue, ROI, cost per acquisition, and break-even conversion rate for email marketing campaigns.
Workflows that use this calculator
Background on the A/B Test Significance Calculator
- Formula Price Elasticity Formula
Uncover customer demand sensitivity to price changes with the Price Elasticity Formula. Essential for optimizing pricing strategies through data-driven.
- Examples A/B Test Significance Examples & P-Values
Four worked A/B test significance examples showing how sample size and lift drive the p-value, so you call a real winner instead of shipping random noise.
- Examples Price Elasticity Examples
Explore practical price elasticity examples across SaaS, retail, and ride-sharing. Learn how demand reacts to price changes and optimize pricing strategies.
- Statistics 15 A/B Testing Statistics
A/B testing statistics on win rates, sample size, and false positives, each taken from a named primary source: Microsoft's experimentation platform and a peer-reviewed review of A/B testing methodology.
- Statistics 15 Marketing ROI Statistics
Marketing ROI statistics on budget share, channel mix, and AI spend, each taken from a named primary source: the Gartner annual CMO Spend Survey of marketing leaders.
- Learn What Is Cart Abandonment Rate? Simply Explained
Understand cart abandonment rate: its definition, formula, real-world impact, and strategies to recover lost sales in e-commerce. Essential for AI Biz Hub.
Articles
- 8 min read How to Run A/B Tests That Actually Work
Run A/B tests that survive replication: pick one metric, power the test, stop peeking, and read results with the practical-significance check most teams skip.
- 9 min read Landing Page Conversion at 2%: The Solo Founder Floor
Landing page conversion at 1.9% on 12,000 monthly visitors, $84 AOV, $640 cost. The page pays only when AOV moves — not when conversion does.
- 7 min read How to Optimize Your Conversion Rate
Optimize conversion rate by prioritising experiments with ICE scores, powering tests to detect realistic lifts, and avoiding the peeking trap.
More in Marketing and acquisition
Know whether your marketing spend is building value or burning cash.
- CAC Calculator
- Churn & Retention Calculator
- Ad Spend / ROAS Calculator
- Email Marketing ROI Calculator
Use it from code
The same calculation as a JavaScript module you can import. It runs where you import it, with no request, key or rate limit.
import { compute } from "https://aibizhub.io/engines/ab-test-significance-calculator.js";Input and output contract, TypeScript types and the guide for agents.