Phase 03: Metrics and product analytics
Product Analytics and Metrics Course: 14 Free Lessons
Define, decompose and diagnose the numbers a business runs on.
This phase is about the numbers a product team runs on: how to break a top-line metric into parts that explain a change, how to count a funnel and active users from raw events without inflating them, and how to pick a north star metric with guardrails and a launch rule, written before the results arrive, that allows for noise. Every lesson builds the metric from raw rows with the Python standard library or SQLite, on a small synthetic dataset, and follows the course format in LESSON_TEMPLATE.md.
Before you start: the SQL lessons in phase 01 and the statistics lessons in phase 02. Each lesson lists its own prerequisites, level (1 beginner, 2 intermediate, 3 advanced) and time.
How to run a lesson: open its folder in a terminal and run python3 test_lesson.py. It fails until you fill in the exercise. CHECK_SOLUTION=1 python3 test_lesson.py runs the tests on the worked answer. Python 3.9 or later, nothing to install.
The 14 lessons, in learning order
- 1. Metric trees
Revenue fell this week, so which part of the business made it fall?
Level 2: Intermediate · about 2 hours
- 2. Funnel conversion
Of the people who look at a course, how many buy it, and at which step do we lose the most of them?
Level 1: Beginner · about 2 hours
- 3. Active users and stickiness
How many people really use the app, and how often do they come back?
Level 1: Beginner · about 2 hours
- 4. North star and guardrails
The new feature moved our main metric up, so should we launch it?
Level 2: Intermediate · about 2 hours
- 5. Define a metric precisely
What exactly counts as an active customer, and what goes in the metric spec?
Level 1: Beginner · about 2 hours
- 6. Percent vs points, ratio of sums vs average of ratios
Did conversion rise 2% or 2 points, and why do two dashboards disagree?
Level 1: Beginner · about 2 hours
- 7. Growth accounting
Where did this month's MAU change come from: new, retained, resurrected or churned users?
Level 2: Intermediate · about 2 hours
- 8. Retention curves and churn
What share of users are still here on day 7 and day 30, and which retention definition fits?
Level 2: Intermediate · about 2 hours
- 9. Segmentation and mix effects
Did conversion fall, or did the traffic mix change?
Level 2: Intermediate · about 2 hours
- 10. Diagnose a metric drop
DAU fell 20% overnight. Tracking bug, holiday, segment, or real?
Level 2: Intermediate · about 2 hours
- 11. Attribution: first touch, last touch, linear
Which channel gets credit for the sale, and how much does the answer depend on the model?
Level 2: Intermediate · about 2 hours
- 12. Unit economics: LTV, CAC and payback
Is paid social acquisition profitable once churn is included?
Level 2: Intermediate · about 2 hours
- 13. Marketplace metrics and liquidity
What share of job posts get an application within 48 hours, and which side is the constraint?
Level 3: Advanced · about 2 hours
- 14. Causal impact without an experiment: difference in differences
We launched in one city and not the others. Did it work?
Level 3: Advanced · about 2 hours
What you will be able to do
By the end of the phase you will have built and tested each of these yourself, one per lesson:
- Metric trees:
build_tree(visitors, orders, revenue): the three branches of the revenue tree. - Funnel conversion:
events_per_step: raw event counts next to distinct user counts, to see the trap. - Active users and stickiness:
active_by_day(path, actions): the set of active users on each day, for a chosen definition of active. - North star and guardrails:
summarize(path): users, applying users, applications, replies and unsubscribes per variant, from raw rows. - Define a metric precisely:
SPEC: the finance sentence as a dictionary of values a test can check. - Percent vs points, ratio of sums vs average of ratios:
ratio_of_sums(rows)andaverage_of_ratios(rows): the finance and the marketing calculation. - Growth accounting:
load_activity(path): each user's set of active months from the raw sessions, with blank ids removed and repeat sessions counted once. - Retention curves and churn:
load(users_path, visits_path): the day offsets of each user's visits, with glitch rows dropped and silent users kept. - Segmentation and mix effects:
load_periods(path, split_date): sessions and orders per device for the prior and the current period. - Diagnose a metric drop:
load(path),totals(rows, date, dims)andpct_change(old, new): the basic counts. - Attribution: first touch, last touch, linear:
parse_time(text)andload(...): one path per purchase, with the 30-day window, touches after the purchase removed and ties in time ordered by channel name. - Unit economics: LTV, CAC and payback:
months_betweenandload(folder): month arithmetic and the three tables. - Marketplace metrics and liquidity:
post_liquidity(posts, applications, as_of, window_hours): filled and eligible posts by city, plus a pooled "all". - Causal impact without an experiment: difference in differences:
cell_meansanddid_from_means: the four averages and the DiD from them.
How to start
Clone the course and run the first lesson's test. It fails until your exercise is right. You need only Python 3.9 or later.
git clone https://github.com/Ani19arya/analytics-from-scratch
cd analytics-from-scratch/phases/03-metrics-and-product-analytics/01-metric-trees
python3 test_lesson.pyJobs that ask for these skills
Open roles on Real Analyst Jobs where this phase's skills come up, read daily from company career pages:
