Phase 08: Modern data stack
Modern Data Stack and dbt for Analysts: 6 Free Lessons
Warehouses, transformation, orchestration, testing and version control: what an analyst must understand and what a data engineer builds.
Where does the data in your dashboard come from, and what has happened to it on the way? This phase follows a number from the source system to the chart: loading, transforming, testing, scheduling and versioning. You rebuild each idea in Python and SQLite, and see the real tools (Snowflake, BigQuery, dbt, Airflow, Git) as reference blocks next to the code you wrote.
The 6 lessons, in learning order
- 1. Warehouse layers: raw, staging and marts
Why do we never edit raw data, and where should a metric be computed?
Level 2: Intermediate · about 2 hours
- 2. dbt from scratch
In what order must these six models build, and what do their tests prove?
Level 2: Intermediate · about 2 hours
- 3. Incremental models and backfills
Can yesterday's numbers be fixed without rebuilding three years of data?
Level 2: Intermediate · about 2 hours
- 4. Orchestration: a tiny DAG runner
What happens at 6 am when the 5 am job failed?
Level 3: Advanced · about 2 hours
- 5. Query cost in a cloud warehouse
Why did one dashboard cost 400 dollars, and how do partitions fix it?
Level 2: Intermediate · about 2 hours
- 6. Git for analysts
Who changed the revenue query, and can we undo it?
Level 1: Beginner · about 2 hours
What you will be able to do
By the end of the phase you will have built and tested each of these yourself, one per lesson:
- Warehouse layers: raw, staging and marts:
stg_orders: one row per order out of 749 raw rows, the latest load wins, dates in one format, money as numbers. - dbt from scratch:
parse_refs,parse_configandcompile_sql: read the template tags out of a model and turn it into SQL that SQLite can run. - Incremental models and backfills:
build_days: rewrite any range of days from the raw rows that had landed by a given date. Everything else is built on it. - Orchestration: a tiny DAG runner:
downstream(dag, job): everything a failure blocks. - Query cost in a cloud warehouse:
block_survives: decide, from a block's metadata alone, whether a query must read it, for a table that is not partitioned, partitioned, or partitioned and clustered. - Git for analysts:
log: the history from HEAD back to the first commit, by following parent pointers.
How to start
Clone the course and run the first lesson's test. It fails until your exercise is right. You need only Python 3.9 or later.
git clone https://github.com/Ani19arya/analytics-from-scratch
cd analytics-from-scratch/phases/08-modern-data-stack/01-warehouse-layers
python3 test_lesson.pyJobs that ask for these skills
Open roles on Real Analyst Jobs where this phase's skills come up, read daily from company career pages:
