A/B test calculator

Tidewater Outdoor tested a shorter checkout page (B) against the current one (A). Dana wants to ship B. Move the numbers and see when a gap is bigger than luck, using the same formulas as the lesson.

1. The two arms

A, current page

visitors

conversions

B, new page

visitors

conversions

2. The test

rate A

rate B

lift (B minus A)

pooled rate

standard error (pooled)

two-sided p-value

95% CI on the lift

The shaded tails are the p-value: the chance of a z at least this far from 0, either way, if the pages were equally good.

sample ratio check, chi-square

sample ratio check, p

The formulas

p_a = conv_a / n_a      p_b = conv_b / n_b      lift = p_b - p_a
p_pool = (conv_a + conv_b) / (n_a + n_b)
SE = sqrt( p_pool * (1 - p_pool) * (1/n_a + 1/n_b) )
z = lift / SE
Phi(z) = 0.5 * (1 + erf(z / sqrt(2)))
p_value = 2 * (1 - Phi(|z|))
SE_diff = sqrt( p_a(1 - p_a)/n_a + p_b(1 - p_b)/n_b )   (unpooled, for the interval)
CI = lift +/- 1.96 * SE_diff
SRM: chi2 = (n_a - E)^2/E + (n_b - E)^2/E with E = (n_a + n_b)/2, p = 2 * (1 - Phi(sqrt(chi2)))
Ship only if p_value < 0.05 and lift > 0. Stop and fix the split if the SRM p is below 0.001.

3. Peeking: why checking every day finds fake winners

Lesson 02-13 simulates A/A tests: both arms have the same true rate, so every "winner" is a false alarm. Each test runs 14 days with 280 users per arm per day at a 12% rate. Peeking means stopping the first day p drops below 0.05.

Tests to simulate Seed

below 0.05 on the last day-

below 0.05 on ANY day-

The chart will show the p-value of the first 20 tests, day by day. Red lines dip below 0.05 at least once.