Sample size scales with 1/effect². Detecting a 5% lift instead of a 10% one does not cost twice as much traffic — it costs four times. That single fact is why most tests are underpowered: the effect people hope for is small, and the cost of proving it is quadratic.
| Baseline | +2% | +5% | +10% | +20% | +50% |
|---|
A test checked repeatedly and stopped the moment p < 0.05 does not have a 5% false positive rate. Every look is another chance for noise to cross the line, and the error rate climbs towards certainty with enough looks. The figures below are the approximate false-positive rate for a test with no real effect at all, computed exactly for the significance level you have set above — equally spaced looks, each judged at that level.
Baseline rate, the effect to detect, how often you accept a false positive, and how often you accept missing a real effect. Fix those and the sample size is determined.
n = (zα/2 + zβ)² × [p₁(1−p₁) + p₂(1−p₂)] ÷ (p₂ − p₁)² per variant. At 5% and 80% power the leading term is 7.85.
A 5% baseline moving to 5.5% is a difference of 0.005; squared it is 0.000025, dividing a numerator near 0.1. That is where 31,000 per variant comes from.
Check daily and stop at the first p < 0.05 and the false-positive rate climbs from 5% toward 30% or more. Fix the horizon, or use a design built for monitoring.
An underpowered test cannot distinguish a real effect from none, so a null result carries almost no information. Put your actual sample into the detectable-effect mode: if the smallest thing it could have found is a 25% lift, the test ruled out nothing anyone cared about. The test did not show the change does not work; it showed the test could not see.
Detecting a 5% lift rather than 10% is four times the traffic, not twice. Teams plan for the effect they hope for and budget for the effect they expected; the gap between those is quadratic.
Check daily and stop the moment p < 0.05 and you are not running a 5% test: every look is a fresh chance for noise to cross the line. The false positive rate climbs toward 30% or more, depending on how often you look. Either fix the sample size in advance and wait, or use a sequential or Bayesian method designed for continuous monitoring.
A convention, not a law, and a generous allowance when the decision is expensive. Going to 90% costs about a third more traffic, usually less than shipping the wrong thing or rerunning the test.
The variance term shrinks in proportion to the rate, but the squared difference shrinks with its square. A 1% conversion rate needs roughly five times the sample of a 5% one for the same relative lift: why small sites cannot A/B test small changes.
Each extra comparison is another shot at a false positive, so the threshold must tighten (Bonferroni divides α by the number of comparisons; this page uses the Šidák level), pushing every arm’s sample up. Running a multi-arm test on a two-arm sample size is one of the commonest ways to generate results that will not replicate.
Most likely because it could not have. "No significant difference" from an underpowered test means the test was too small to see the effect, not that the effect is absent. Put the sample you actually had into the detectable-effect mode: if the smallest thing you could have found is a 25% lift, a null result rules out almost nothing worth knowing.
No — not with a fixed-horizon test. Checking daily and stopping at the first p < 0.05 pushes the false positive rate from 5% towards 30% or more depending on how often you look. Either fix the sample size in advance and wait, or use a method designed for continuous monitoring such as a sequential test or a Bayesian approach.
It is the convention, and it means accepting a one in five chance of missing a real effect. That is a lot for an expensive decision. Raising power to 90% costs about 34% more traffic, which is usually cheaper than shipping the wrong thing or rerunning the test.
Each additional comparison is another chance of a false positive, so the significance threshold has to tighten — a Bonferroni correction divides α by the number of comparisons, which increases the sample needed for each. Four variants tested against control is not one test, it is three, and pretending otherwise inflates the error rate.
Two independent groups, one binary outcome, random assignment, a fixed sample size decided before the test starts, and a normal approximation to the binomial — reliable when the expected count in each cell is above about five. Multiple comparisons across variants are corrected with Šidák, and a finished test can be read with a Newcombe interval on the lift rather than a bare p-value. Not modelled: sequential or Bayesian designs, novelty and primacy effects, day-of-week and seasonal variation, users appearing in both arms, non-binary metrics like revenue per user, or cluster effects when the unit assigned is not the unit measured.
Releases in which this page changed, newest last. Derived from the archived copy of every release, not from notes written afterwards — so it reflects what actually shipped. Site-wide passes are left out; they are in the full changelog.