Phase 02: Statistics and experiments

Statistics and A/B Testing for Analysts: 15 Free Lessons

Probability, sampling, tests and regression by hand, so you know what a p-value or an interval really says.

15 lessonsAbout 30 hours5 beginner, 7 intermediate, 3 advancedFree, runs offline

This phase teaches the statistics an analyst uses every week: how to summarise a skewed number honestly, how sure you can be about an average or a rate from a sample, whether an A/B test result is real or luck, and how big a test must be before it starts. Every method is built by hand with the Python standard library (no statistics module, no scipy), on a small synthetic dataset, and checked by a test that computes the answer independently. Each lesson follows the course format in LESSON_TEMPLATE.md: a business problem, the idea in plain words, a diagram, a walk through, common mistakes, an exercise and what to tell the stakeholder.

You need Python 3.9 or later. No SQL is required, though GROUP BY from raw rows helps with lesson 01.

The 15 lessons, in learning order

1. Probability and distributions by simulation
If 10% of visitors buy, how many of 200 should buy, and how unusual is 30?
Level 1: Beginner · about 2 hours
2. Mean, median and skew
What is the typical posted pay for an analyst role, and which number should we put on the slide?
Level 1: Beginner · about 2 hours
3. Outliers, box plots and robust summaries
Which salaries are real outliers, and which summary survives them?
Level 1: Beginner · about 2 hours
4. Sampling and the central limit theorem
Why does the average of 18 orders wobble, and by how much?
Level 1: Beginner · about 2 hours
5. Confidence interval by hand
How sure are we about an average or a rate measured on a sample?
Level 1: Beginner · about 2 hours
6. Hypothesis testing by permutation
Could this gap appear just by shuffling the labels?
Level 2: Intermediate · about 2 hours
7. A/B test from scratch
Did the new checkout page really convert better, or did we get lucky?
Level 2: Intermediate · about 2 hours
8. Sample size before you test
How many users, and how many weeks, do we need before an A/B test can answer the question we care about?
Level 2: Intermediate · about 2 hours
9. Comparing categories: the chi-square test
Do the three signup flows differ in completion rate?
Level 2: Intermediate · about 2 hours
10. Correlation, causation and Simpson's paradox
Users who get more notifications work out more. Should we send more?
Level 2: Intermediate · about 2 hours
11. Linear regression from scratch
How much does each extra visit add to monthly spend, and what does R squared really say?
Level 2: Intermediate · about 2 hours
12. Bootstrap confidence intervals
How sure are we about a median or a ratio when no simple formula exists?
Level 2: Intermediate · about 2 hours
13. A/B test pitfalls
We tested five variants on three metrics. Which wins are real?
Level 3: Advanced · about 2 hours
14. Ratio metrics and the delta method
Is the drop in replies per application real when one user sends many?
Level 3: Advanced · about 2 hours
15. Variance reduction with CUPED
Can pre-test behaviour make the test finish in half the time?
Level 3: Advanced · about 2 hours

What you will be able to do

By the end of the phase you will have built and tested each of these yourself, one per lesson:

Probability and distributions by simulation: count_buyers(path): visitors and buyers from the raw rows.
Mean, median and skew: mean(values) and median(values), with no statistics module.
Outliers, box plots and robust summaries: percentile, median and quartiles, by the inclusive method, written down so anyone can reproduce your numbers.
Sampling and the central limit theorem: mean, population_sd (divide by N) and sample_sd (divide by n - 1).
Confidence interval by hand: sample_sd(values): the sample standard deviation with the n - 1 denominator.
Hypothesis testing by permutation: read_groups(path): order values per group, skipping blank values instead of reading them as 0.
A/B test from scratch: normal_cdf(z): the standard normal CDF using math.erf.
Sample size before you test: z_quantile(p): the inverse of the normal CDF, found by bisection.
Comparing categories: the chi-square test: read_table(path): a contingency table from raw rows, skipping visitors whose outcome is not known yet.
Correlation, causation and Simpson's paradox: read_users(path): users with a known workout count, skipping the ones who turned tracking off.
Linear regression from scratch: read_members(path): visits and spend, skipping members whose spend is unknown.
Bootstrap confidence intervals: read_orders(path) and read_sessions(path): order values, and sessions both one row each and rolled up per user.
A/B test pitfalls: chi2_sf(x, df): the chi-square tail probability for any whole number of degrees of freedom, with no library.
Ratio metrics and the delta method: read_users(path): one (applications, replies) pair per user, by variant.
Variance reduction with CUPED: read_users(path) and fill_missing(xs): the data, with a rule for users who have no pre-period.

How to start

Clone the course and run the first lesson's test. It fails until your exercise is right. You need only Python 3.9 or later.

git clone https://github.com/Ani19arya/analytics-from-scratch
cd analytics-from-scratch/phases/02-statistics-and-experiments/05-probability-by-simulation
python3 test_lesson.py

This phase on GitHub

Jobs that ask for these skills

Open roles on Real Analyst Jobs where this phase's skills come up, read daily from company career pages:

All 12 phases of Analytics from Scratch