A/B Test Calculator

What the p-value actually means

It is the probability of seeing a gap at least this large if the two versions were really identical. A p-value of 0.03 does not mean there is a 97% chance B is better; it means a difference this big would turn up 3% of the time by luck alone. That distinction is the one most often mangled in a readout to stakeholders.

Why the confidence interval matters more

“Significant” is a yes/no answer to a question nobody asked. The interval tells you the size of the effect you can actually defend: a lift somewhere between +0.2 and +5.8 percentage points is a very different business case from a lift of exactly +3. This calculator uses the unpooled standard error for the interval and the pooled one for the test, which is the textbook pairing and the reason the two never contradict each other here.

The mistake that ruins most tests

Watching the dashboard and stopping the moment it goes green. If you check repeatedly and stop at the first significant reading, you will hit “significant” on tests where nothing is happening: the 5% false positive rate applies to one look, not to twenty. Decide the sample size before you start, then look once.

What this tool does not do

It compares two conversion rates: visitors in, conversions out. It cannot test revenue per user, session length or anything else measured on a continuous scale, because those need the spread of the underlying values and not just two totals. If someone hands you an average order value and asks whether it moved, this is the wrong instrument.