Sample size before you test: Free Analytics Lesson

The problem

Lumen Learning Co, a fictional online course platform, wants to run tests on six pages this quarter. Sam, the growth lead, asks for a plan by the end of the week: which pages can be tested in a reasonable time, and which would take months. The team has been stopping tests "when the result looks clear", which is how they shipped two changes last year that later showed no effect. You will compute the required sample size from the formula, before any test starts, and turn it into weeks of traffic.

The file data/pages.csv is synthetic, generated by make_data.py with a fixed seed. It holds a baseline conversion rate and weekly visitors for each page.

What you will build from scratch

z_quantile(p): the inverse of the normal CDF, found by bisection.
sample_size_per_group(p1, p2, alpha, power): users needed per group.
sample_size_for_lift(baseline, relative_lift, ...): the same, from a relative lift.
weeks_needed(n_per_group, weekly_visitors): turns users into calendar time.
plan(path, relative_lift): the per-page plan for Sam.

The idea

A test can go wrong in two ways:

False positive (type I error): you declare a difference that is not real. Its rate is alpha, usually 0.05.
False negative (type II error): a real difference exists but the test misses it. Its rate is beta. Power = 1 - beta, usually 0.80.

Before a test you choose alpha, power and the smallest effect worth detecting, often called the minimum detectable effect (MDE). The sample size follows from those three choices and the baseline rate.

For two proportions, baseline p1 and target p2, with a two-sided test:

z_a   = z value with 1 - alpha/2 of the normal below it   (1.96 for alpha 0.05)
z_b   = z value with power of the normal below it          (0.8416 for power 0.80)
p_bar = (p1 + p2) / 2

n per group = ( z_a * sqrt(2 * p_bar * (1 - p_bar)) + z_b * sqrt(p1*(1-p1) + p2*(1-p2)) )^2
              / (p2 - p1)^2

Read the formula from the bottom. The difference (p2 - p1) is squared in the denominator. Halve the effect you want to detect and you need about four times as many users. That is the whole reason small lifts are expensive.

The z values come from the inverse normal CDF. There is no closed form, but the CDF only goes up, so bisection works: guess the middle of an interval, check whether the CDF there is below or above the target, keep the half that contains the answer, and repeat. Each step halves the error.

Walk through

Inverse normal by bisection:

import math
def cdf(z):
    return 0.5 * (1 + math.erf(z / math.sqrt(2)))
low, high = -10.0, 10.0
for _ in range(100):
    mid = (low + high) / 2
    if cdf(mid) < 0.975:
        low = mid
    else:
        high = mid
print(round(mid, 4))   # 1.96

Sample size for 10% to 12% (a 20% relative lift), alpha 0.05, power 0.80:

import math
p1, p2, za, zb = 0.10, 0.12, 1.959964, 0.841621
p_bar = (p1 + p2) / 2
top = (za * math.sqrt(2 * p_bar * (1 - p_bar)) + zb * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
print(math.ceil(top / (p2 - p1) ** 2))   # 3841 per group

Now the plan. On the lesson data, python3 solution/solution.py gives these users per group:

page	baseline	20% lift	10% lift	5% lift
search	35.8%	730	2,873	11,380
checkout	17.0%	2,064	7,964	31,262
home	5.3%	7,670	29,368	114,838
signup	2.8%	14,942	57,135	223,262

Two patterns stand out. Each halving of the lift multiplies the sample by roughly four. And low baseline rates need far more users for the same relative lift, because the absolute difference is tiny: 10% of 2.8% is 0.28 percentage points.

Traffic turns this into time. The cart page has a 5.6% baseline and 10,500 visitors a week. Detecting a 10% lift needs 27,702 per group, which is 6 weeks of all its traffic. A 5% lift needs 21 weeks, which is probably not worth running. The search page can detect a 5% lift in a single week.

What to tell Sam: "Test search, checkout and home first. For cart, only test changes we expect to move conversion by 20% or more, otherwise the test will run for months. Fix the sample size before each test and read the result once, at the end."

Your turn

Open exercise.py and fill in the functions marked TODO. normal_cdf is given. Then run, from this folder:

python3 test_lesson.py

To see the reference solution pass: CHECK_SOLUTION=1 python3 test_lesson.py.

Check your numbers with the sample size calculator. Small differences of a user or two come from rounding in the z values.

Check yourself

A stakeholder wants to detect a 2% relative lift on a page with a 3% baseline. What do you tell them?
What happens to the required sample size if you raise power from 0.80 to 0.90?
Why should the sample size be fixed before the test starts?

Answers

The absolute effect is 0.06 percentage points, and the squared difference in the denominator makes the required sample enormous. Ask whether a bigger change is possible, whether a higher-traffic or higher-baseline metric could be used, or whether the decision can be made without a test.
It goes up, because z_b rises from about 0.84 to about 1.28. You are asking to miss a real effect less often, which costs more users.
If you keep checking and stop when the result looks good, the real false positive rate is much higher than alpha. A fixed sample size, set from alpha, power and the MDE, keeps the error rates you chose.

In real job postings

On 25 October 2026 the Real Analyst Jobs live feed had 1,193 live analyst postings in India; of the 914 that list tools, 49% ask for Python and 70% ask for SQL, the two tools you need to pull traffic numbers and run a plan like this one. See the India analyst job market report or browse roles at Real Analyst Jobs.

Phase 02 is done. Go on to Phase 03: Metrics and product analytics, starting with Metric trees.

Sample size before you test