Skip to content

A/B Test Sample Size Calculator

The minimum sample per variant to detect a given lift at your chosen confidence level. So your tests don’t end on noise.

Test parameters

Why A/B tests fail

The number one reason an A/B test “doesn’t conclude anything” isn’t that the variant doesn’t work: it’s that you didn’t have a big enough sample to detect it. If your baseline conversion is 2% and you want to detect a 10% relative lift (from 2% to 2.2%), you need around 15,000 visitors per variant at 95% confidence and 80% power. If you stop the test at 2,000 visitors, what you see is noise, not signal.

The other usual reason: peeking. Checking the test every day and stopping when it’s “already winning” multiplies false positives. If you set the duration upfront and stick to it, the false-positive risk stays at the confidence level you chose. If you stop early, the real risk can climb to 30-50%.

Three kinds of tests by sample needed

  • • Big tests (MDE 30%+): radical visual changes, new products. Few visitors needed (1,000-5,000 per variant). Good for low-traffic sites.
  • • Medium tests (MDE 10-20%): copy, layout and CTA improvements. Moderate needs (10,000-50,000 per variant). The sweet spot for everyday CRO.
  • • Fine-grained tests (MDE 3-5%): marginal optimization on an already polished site. Tens of thousands per variant. Only viable with massive traffic.

When an A/B test isn’t worth it

  • • Very low traffic (<5,000 visitors a month on the tested page).
  • • A change that will ship whether it wins or loses (for legal, technical or brand reasons).
  • • The change is obvious: a full brand redesign, a new business model.
  • • The cost of not testing is very low (an internal page with minimal traffic).
  • • You don’t have time: if you need a decision in 2 days, use other methods (heuristics, usability).

Common mistakes

  1. Stopping the test when it’s “already winning”. False positives shoot up to 30-50%.
  2. Not running full cycles. At least 2 weeks to capture several Monday-to-Sunday cycles.
  3. Testing too much at once with low traffic: every test dilutes the sample.
  4. Ignoring instrumentation: if tracking breaks, the result is garbage.
  5. Assuming a winner in one segment wins in all of them. Segment your results.

What to do with the result

When your test ends, don’t roll out every winning variant at random: prioritize by absolute revenue impact. A variant that lifts conversion 3% on a page with 1M visits a month is worth more than one that lifts it 20% on a page with 10k. Also, document what you learned, not just what won: knowing why is what scales to your next tests.

Keep going