A/B Test Calculator
Calculate statistical significance and determine the winner of an A/B test using Z-score and normal approximation.
Did this tool work for you?
How to use this calculator
Conversion rates for A and B are calculated, then pooled. The standard error accounts for both sample sizes. A Z-score above 1.645 gives 95% one-tail significance; above 1.96 gives 97.5%.
- 1
Enter the number of visitors and conversions for your control (Variant A) and test (Variant B).
- 2
The calculator computes conversion rates, relative uplift, and Z-score automatically.
- 3
Check Statistical Significance — 95% or higher means the result is unlikely due to chance.
- 4
Only call a winner once significance is 95%+; end the test early and you risk a false positive.
Frequently asked questions
Which statistical test does this calculator run?
A two-proportion z-test with a pooled standard error. It computes each conversion rate (conversions ÷ visitors), pools both groups into one rate p = (cA + cB) ÷ (nA + nB), takes SE = √(p × (1 − p) × (1/nA + 1/nB)) and reports Z = (pB − pA) ÷ SE. The Z-score is then converted to a probability with a normal-curve approximation. There is no exact test, continuity correction or Bayesian model behind it.
How is significance decided and a winner declared?
Significance is the normal CDF of the absolute Z-score, shown as a percentage. That is a one-tailed figure: it asks how likely a gap this large would be if the leading variant were really no better. A winner is announced when it reaches 95%, which means |Z| ≥ 1.645, and the winner is B when Z is positive and A when Z is negative. Below 95% the result reads “No significant winner yet”.
Is 95% here the same as p < 0.05?
Not quite. Because the figure is one-tailed, 95% corresponds to a two-tailed p-value of about 0.10. The default inputs show this: Z = 1.9451, significance 97.4%, so Variant B is declared the winner, yet the two-tailed p-value is roughly 0.052. If you want the stricter two-tailed 5% convention, wait until the page shows 97.5% or more, which is |Z| ≥ 1.96.
Does 97% significance mean there is a 97% chance that B is better?
No. The number is a transformed Z-score, not the probability that a variant is better, and it is not the chance that the result came from luck either. It only describes how surprising the observed gap would be if both versions truly converted at the same rate. Estimating the probability that B beats A needs a Bayesian model, which this page does not use.
How is relative uplift calculated, and why is it not in percentage points?
Uplift is (rate B − rate A) ÷ rate A × 100. With 3.00% for A and 3.70% for B the gap is 0.70 percentage points but the relative uplift is 23.33%. If A’s rate is zero the uplift is shown as 0.00% because the ratio is undefined. Uplift is a point estimate from this sample and carries no interval.
What happens with zero conversions or impossible inputs?
Visitor counts below 1 are treated as 1, negative conversions become 0, and conversions larger than visitors are capped at the visitor count so a rate can never exceed 100%. If both groups have zero conversions, or every visitor converted, the standard error is zero, Z is reported as 0 and significance shows 50.0%, so no winner is named.
Can I check the result every day and stop when it turns significant?
That is a known way to get false positives. The 95% threshold assumes you look once, at a sample size fixed in advance; repeated looks give random noise many chances to cross the line. Running several variants or reading many metrics without adjusting the threshold has a similar effect. This calculator applies no correction for either, so fix the sample size before you start.
What does it not do?
It does not compute required sample size or statistical power, confidence intervals, or revenue-per-visitor comparisons, and it handles exactly two variants. The normal approximation is also weak when conversions are very scarce; a common rule of thumb is to want roughly ten or more conversions and ten or more non-conversions in each group before trusting a z-test.
A/B test significance: a pooled two-proportion z-test, step by step
Pooled standard error and the Z-score
The calculator assumes the null hypothesis that both variants share one true conversion rate, so it pools the two samples into p = (cA + cB) ÷ (nA + nB). The standard error is √(p × (1 − p) × (1/nA + 1/nB)), and Z is the observed gap pB − pA divided by that error. Z counts how many standard errors apart the two rates sit. The tool converts it with a polynomial approximation of the normal CDF, which is accurate to several decimal places for this purpose, then multiplies by 100 to display significance.
Worked example: the default test
Variant A converts 150 of 5,000 visitors (3.00%) and Variant B converts 185 of 5,000 (3.70%). The relative uplift is 23.33%, Z is 1.9451 and significance is 97.4%, so the tool announces “Variant B wins”. Swap the groups, so A has 130 of 1,000 and B has 100 of 1,000, and the sign flips: Z becomes −2.1027, significance 98.2% and the tool names Variant A the winner. The magnitude is judged either way, and only the sign of Z picks the side.
What different results look like
The table lists runs of the tool with fixed inputs. The third row is a smaller test: a 25% relative uplift sounds large, but 48 against 60 conversions out of 2,400 each gives Z of only 1.1679, short of the 95% bar. The last row shows the zero-conversion edge case.
| Visitors A / B | Conversions A / B | Z-score | Significance | Outcome |
|---|---|---|---|---|
| 5,000 / 5,000 | 150 / 185 | 1.9451 | 97.4% | Variant B wins |
| 1,000 / 1,000 | 100 / 130 | 2.1027 | 98.2% | Variant B wins |
| 1,000 / 1,000 | 130 / 100 | -2.1027 | 98.2% | Variant A wins |
| 2,400 / 2,400 | 48 / 60 | 1.1679 | 87.9% | No significant winner yet |
| 1,000 / 1,000 | 0 / 0 | 0.0000 | 50.0% | No significant winner yet |
Reading one-tailed against two-tailed
For a standard normal curve, Z = 1.645 leaves 5% in one tail and Z = 1.96 leaves 2.5% in each of two tails. This page declares a winner at the first of those cut-offs, which suits a question asked in one direction before the test started. If you would have acted on a loss for B as well, the two-tailed cut-off of 1.96 (97.5% on this page) is the fairer bar. Decide which question you are asking before you look at the numbers.
Limits worth knowing before you ship a variant
A significant result says the gap is unlikely to be pure noise; it does not say the gap is large, that it will persist, or that the test ran long enough to cover weekday and weekend behaviour. Fix the sample size and the stopping rule in advance, split traffic randomly, and treat a surprising uplift from a small sample with suspicion. The result also covers only the single metric you entered, not downstream effects such as refunds or retention.
Learn more from an authoritative source:
WikipediaInfluencer ROI Calculator
Calculate the return on investment of influencer marketing campaigns including cost per impression, engagement, and click.
Content Marketing ROI Calculator
Calculate the return on investment of your content marketing programme including cost per lead, cost per customer, and total ROI.
Ad Budget Calculator
Calculate the required ad spend to hit a revenue target based on ROAS, average order value, and conversion rate.
Engagement Rate Calculator
Calculate social media engagement rate for any platform and compare it against industry benchmarks.
Funnel Conversion Calculator
Calculate conversion rates at every stage of your marketing and sales funnel — from visitor to customer.
Keyword Traffic Value Calculator
Estimate the monetary value of your organic keyword traffic and potential revenue compared to equivalent paid search spend.
Results are estimates for informational purposes only and do not constitute professional financial, medical, legal, or technical advice. Read full disclaimer →