North star and guardrails

The problem

Jobnest, a fictional job marketplace, tested a "quick apply" button that sends a saved profile to an employer in one tap. 1,800 job seekers were split at random between the current flow (control) and quick apply for two weeks. Lena, the product director, sees more people applying and wants to launch on Monday. The employer team is worried that one-tap applications are lower quality and that employers will stop replying. You will define the north star and two guardrails, compute all three from the raw data, and apply a launch rule that was written before anyone looked at the results.

The file data/experiment.csv is synthetic, generated by make_data.py with a fixed seed. Each row is one made-up job seeker: variant, applications sent, employer replies received and whether they unsubscribed from emails.

What you will build from scratch

• summarize(path): users, applying users, applications, replies and unsubscribes per variant, from raw rows.
• metrics(summary): the north star and the two guardrail metrics.
• decide(control, treatment, ...): the launch rule, as code, with its tolerances as parameters.

The idea

A north star metric is the one number the team agrees best captures the value the product delivers, so that if it grows, the business grows for the right reasons. A good one:

1. Measures value delivered to the customer, not activity the company can inflate.
2. Is a rate or count of users, so it can move week by week and be compared across tests.
3. Leads revenue instead of being revenue, so teams can act on it.

For Jobnest the team picked:

north_star = applying users / users     (share of job seekers who send at least one application)

No single number is safe on its own. Any metric you push hard can be gamed, by the product as much as by people. Guardrail metrics are the things that must not get worse while you push the north star. A marketplace has two sides, so pick one guardrail per side:

reply_rate       = employer replies / applications   (are employers still getting applications worth answering?)
unsubscribe_rate = unsubscribed users / users        (are we annoying job seekers?)

The decision rule is written before the test, with numbers:

launch only if
    north star lift            >= +2.0 points
    and reply_rate drop        <=  2.0 points
    and unsubscribe_rate rise  <=  0.5 points
otherwise do not launch

All changes are absolute (treatment minus control, in percentage points). A guardrail is a veto: a breach blocks the launch however much the north star went up. That is the point of writing the rule first. After the results are in, a big north star win always makes a guardrail breach look acceptable, and a rule chosen afterwards is a rule chosen to fit the answer.

A breach does not mean the idea is dead. It means this version does not ship, and the team either fixes the cause and tests again or makes the trade explicitly, with the people who own the guardrail in the room.

Walk through

Step 1. Summarize the raw rows per variant.

import csv
out = {}
with open("data/experiment.csv", newline="") as f:
    for row in csv.DictReader(f):
        s = out.setdefault(row["variant"], [0, 0, 0, 0, 0])
        apps = int(row["applications"])
        s[0] += 1
        s[1] += 1 if apps > 0 else 0
        s[2] += apps
        s[3] += int(row["replies"])
        s[4] += int(row["unsubscribed"])
print(out)
# {'control': [889, 269, 658, 133, 21], 'quick_apply': [911, 333, 1190, 156, 16]}

Step 2. The three metrics, from python3 solution/solution.py:

Step 3. Is the north star lift real? Use the two-proportion z-test from phase 02.

import math
ca, na, cb, nb = 269, 889, 333, 911
pool = (ca + cb) / (na + nb)
se = math.sqrt(pool * (1 - pool) * (1 / na + 1 / nb))
z = (cb / nb - ca / na) / se
p = math.erfc(abs(z) / math.sqrt(2))
print(round(z, 2), round(p, 4))   # 2.83 0.0047

The lift is unlikely to be noise. (The reply rate is per application, and one user sends several, so a plain z-test on it would overstate the precision. A 7-point drop against a 2-point tolerance is a breach either way.)

Step 4. Apply the rule. The north star passes (+6.29 against +2.0) and unsubscribes went down, but the reply rate fell 7.10 points against a tolerance of 2.0. Decision: do not launch.

Look at why. Quick apply users sent 1,190 applications against 658 in control, but employers replied to 156 of them against 133. Employers got 81% more applications and sent 17% more replies, so most of the extra applications were not worth answering. Over time, employers who stop replying make the marketplace worse for every job seeker.

What to tell Lena: "Quick apply raised the share of people applying from 30.3% to 36.6%, and that lift is real. It also dropped the employer reply rate from 20.2% to 13.1%, far past the 2-point limit we agreed before the test, so by our own rule it does not ship. The extra applications are mostly ones employers ignore. Next step: test a version that asks one question about fit before the tap, and check the reply rate again."

Your turn

Open exercise.py and fill in the three functions using only csv. Then run, from this folder:

python3 test_lesson.py

To see the reference solution pass: CHECK_SOLUTION=1 python3 test_lesson.py.

Check yourself

1. What makes a good north star metric? Why not use revenue?
2. Why write the decision rule before the test starts?
3. The north star rose 6 points and a guardrail fell 7. The CEO says ship it anyway. What do you say?
4. Suggest a north star for Jobnest that would not have rewarded this launch.

In real job postings

In the Real Analyst Jobs live feed on 25 October 2026, 24% of the 2,102 US analyst postings that list tools ask for Power BI and 36% ask for Tableau, the tools where a north star and its guardrails usually end up on one dashboard. See Power BI for analyst jobs and the US analyst job market report.

Next

Phase 04: Dashboards