The Winning Variant
An A/B test on a new pricing screen shows the new variant converting far better, and growth wants to ship it to everyone. Check whether the win is real before it goes live — and give the call on shipping it.
The brief
You have joined Cadence, a subscription running-coach app, as an analyst.
Your manager forwards a thread from the growth review:
"We tested a new pricing screen last month. Variant B is converting visitors to paid at nearly 12% against about 9% on the old screen — call it a 3-point lift. The plan is to ship B to 100% of traffic on Monday. Can someone give it a final look before we flip it?"
Nobody has looked past the single headline number. You have the raw experiment logs for the whole of June.
What you are asked to do: establish whether B actually converts better, and give a straight recommendation on shipping it.
All timestamps in the data are UTC, and there is no other timezone to reconcile. The experiment window is the calendar month of June 2026 — 1 June 00:00:00 to 30 June 23:59:59 UTC.
Three notes from the team, worth reading before you start:
The bucketing service was redeployed midway through the test. Nobody confirmed afterwards that the traffic split had held.
A visitor who returns during the test can be written a fresh assignment row. Treat a visitor by the first assignment they were given.
The conversions export lists every subscription start it has on record for these visitors — not only the ones inside the test window.
Treat this as you would any experiment readout: check that the two groups are actually comparable before you trust a difference between them, and be ready to show how you got every figure you report.
What to submit
- 1Your five verified answers, entered on the submission form.
- 2A one-page recommendation in your repo README — whether the B result is real, how you know, and whether to ship it. One page, not five.
- 3The SQL, Python or R you used to get your five answers, in the same repo.
- 4Optional: a chart, a dashboard link, or a three-minute Loom. None of these are required and none of them earn marks on their own.
How it is scored
| Area | Weight |
|---|---|
Data accuracyauto-graded The five verified answers. Graded automatically against the dataset, so this half of your score is arithmetic, not opinion. | 40 |
Problem framingreviewed Did you check that the two groups were comparable before trusting the headline lift, rather than confirming it? Restating the brief scores low. | 15 |
Analysis and methodreviewed Did you segment by device and standardise the mix, and did you decide explicitly which rows and which conversions belong in the test? Correct numbers with no visible method score low. | 20 |
Insight qualityreviewed Naming the reversal is not the finding. Why the aggregate and the segments disagree — the broken split — with evidence, is. | 15 |
Recommendationreviewed A clear answer on shipping B, argued from your own numbers, plus what you would do instead. Hedging scores low. | 10 |
The data
Two files, about 4 MB total.
assignments.csv — one row per assignment event. Columns: visitor_id, variant, assigned_at, device, channel, country. A visitor can appear more than once; count each visitor under their earliest assigned_at.
conversions.csv — one row per recorded subscription start. Columns: visitor_id, converted_at, plan, amount_usd. A conversion belongs to the test only if it happened on or after that visitor's first assignment and on or before the end of the window.
device is either web or mobile, written with inconsistent capitalisation. A small share of country values are blank. Neither is a trap; neither changes an answer. The data is synthetic, generated for this challenge, and contains the kind of mess real experiment logs contain. No visitor in it is a real person.
Recognition
Free to enter — everyone is scored and ranked
Every entrant is graded against the same rubric, ranked on the public leaderboard, and gets written feedback on their analysis — no fee, no prize, just the credential.
- The full brief and dataset
- Your score on the verified answers
- Written feedback on your analysis
- Ranked on the public leaderboard
- A certificate and badges you can add to your CV
Rules
- Scoring follows the published rubric below: 40% verified answers, graded automatically, and 60% your recommendation, scored against the rubric
- Scoring is automated. The top 10 submissions and any disputed score are reviewed by a human before results are finalized
- Your repository must be public and its commit history must show the work happening before the deadline
- Work individually. You may use any tool, including AI assistants — but you must be able to explain every number you submit
- One submission per person. You can update it any time before the deadline
- Everyone who submits is ranked by name on the public leaderboard when results are published
- Your repo has to be public for us to read it — but do not promote your solution on LinkedIn, X or Discord until judging closes, so others can still attempt it
