A/B test from scratch

The problem

Tidewater Outdoor, a fictional camping gear shop, tested a shorter checkout page against the current one. 1,900 visitors were split at random between variant A (current) and variant B (new). Dana, the product manager, sees B ahead and wants to ship it in Thursday's release, but the engineering lead asks whether the gap is bigger than random noise. You will write the two-proportion z-test yourself, line by line, and give Dana a ship or do not ship answer with the reasoning.

The file data/experiment.csv is synthetic, generated by make_data.py with a fixed seed. Each row is one made-up user.

What you will build from scratch

• normal_cdf(z): the standard normal CDF using math.erf.
• count_conversions(path): conversions and users per variant from the raw rows.
• two_proportion_ztest(...): pooled rate, standard error, z and the two-sided p-value.
• decide(result, alpha): the ship rule.

The idea

Start with the null hypothesis: A and B have the same true conversion rate. If that is true, any gap we see is noise. The test asks how surprising our gap would be under that assumption.

Step 1. The two observed rates:

p_a = conv_a / n_a
p_b = conv_b / n_b
lift = p_b - p_a

Step 2. Under the null both groups share one rate, so estimate it from everyone:

p_pool = (conv_a + conv_b) / (n_a + n_b)

Step 3. The standard error of the difference between two independent proportions, using the pooled rate:

SE = sqrt( p_pool * (1 - p_pool) * (1/n_a + 1/n_b) )

Step 4. How many standard errors away from zero is the lift?

z = lift / SE

Step 5. The two-sided p-value is the chance of a z at least this far from zero in either direction if the null were true. The standard normal CDF can be written with the error function:

Phi(z) = 0.5 * (1 + erf(z / sqrt(2)))
p_value = 2 * (1 - Phi(|z|))

Step 6. Decide before you look. Pick alpha (0.05 is the common default). Ship only if p_value < alpha and the lift is in the direction you wanted. A significant drop is a reason not to ship, not a win.

The test assumes users are assigned at random, each user counts once, and both groups are large enough that the normal approximation holds (a rough check: at least 10 conversions and 10 non-conversions in each group).

Walk through

The CDF from erf:

import math
def normal_cdf(z):
    return 0.5 * (1 + math.erf(z / math.sqrt(2)))
print(round(normal_cdf(1.96), 4))   # 0.975

A worked example: 200 of 2,000 convert on A, 260 of 2,000 on B.

import math
ca, na, cb, nb = 200, 2000, 260, 2000
lift = cb / nb - ca / na                                   # 0.03
pool = (ca + cb) / (na + nb)                               # 0.115
se = math.sqrt(pool * (1 - pool) * (1 / na + 1 / nb))      # 0.01009
z = lift / se                                              # 2.974
p = 2 * (1 - 0.5 * (1 + math.erf(abs(z) / math.sqrt(2))))
print(round(z, 3), round(p, 4))                            # 2.974 0.0029

Counting from raw rows, one pass over the file:

import csv
counts = {"A": [0, 0], "B": [0, 0]}
with open("data/experiment.csv", newline="") as f:
    for row in csv.DictReader(f):
        counts[row["variant"]][0] += int(row["converted"])
        counts[row["variant"]][1] += 1
print(counts)   # {'A': [107, 976], 'B': [135, 924]}

On the lesson data, python3 solution/solution.py reports A at 10.96% and B at 14.61%, a lift of 3.65 percentage points. The pooled rate is 12.74%, the standard error is 0.0153, z is 2.38 and the two-sided p-value is 0.017. That is below 0.05 and the lift is positive, so the decision is ship.

What to tell Dana: "B converted at 14.6% against 11.0% for A. A gap this large would show up about 1.7% of the time if the pages were equally good, so we are confident B is better. The true lift could still be smaller than 3.6 points; a confidence interval on the difference would show how much smaller."

Your turn

Open exercise.py and fill in the four functions using only math and csv. Then run, from this folder:

python3 test_lesson.py

To see the reference solution pass: CHECK_SOLUTION=1 python3 test_lesson.py.

Check your answer with the A/B test calculator by entering the visitors and conversions for A and B.

Check yourself

1. Why do we pool the two rates when computing the standard error for the test?
2. The p-value is 0.017. Does that mean there is a 1.7% chance B is not better?
3. The test ran for two days, you checked results every hour, and stopped the first time p dropped below 0.05. What is wrong?
4. B is significantly better but the lift is 0.1 percentage points. Do you ship?

In real job postings

In the Real Analyst Jobs live feed on 25 October 2026, 72% of the 2,102 US analyst postings that list tools ask for SQL, which is usually how the experiment counts above get pulled before the test is run. See SQL for analyst jobs and try the SQL case study.

Next

04: Sample size before you test