AI for Data Analysts: 15 Free Lessons on LLMs and Agents

1. 1. Grade an LLM classifier

   Is the LLM good enough at spotting refund tickets that we can route them automatically?

   Level 2: Intermediate · about 2 hours

2. 2. Check LLM-written SQL

   Which of the queries the LLM wrote can we trust, and which ones would have put a wrong number in the report?

   Level 2: Intermediate · about 2 hours

3. 3. Catch made-up numbers

   Every number in the AI-written summary: is it actually in the data?

   Level 2: Intermediate · about 2 hours

4. 4. How LLMs work, for analysts

   What is a token, what does a run cost, and why does the same prompt give different answers?

   Level 1: Beginner · about 2 hours

5. 5. Prompt versions as experiments

   Is prompt v2 better than v1 on the same 200 tickets?

   Level 1: Beginner · about 2 hours

6. 6. Structured output you can load

   Can we put the model's JSON straight into a table?

   Level 2: Intermediate · about 2 hours

7. 7. How big must an eval set be?

   How many labelled examples do we need before we trust the score?

   Level 2: Intermediate · about 2 hours

8. 8. Human in the loop: confidence thresholds

   How many tickets can we auto-route at 95% precision, and who reviews the rest?

   Level 2: Intermediate · about 2 hours

9. 9. LLM as judge, and judging the judge

   Can a model grade answers, and does its grading agree with people?

   Level 2: Intermediate · about 2 hours

10. 10. Natural language to SQL, safely

    Can we let people ask the database questions in English without breaking anything?

    Level 2: Intermediate · about 2 hours

11. 11. Keep personal data out of prompts

    Is any customer's email or phone number about to leave the building?

    Level 1: Beginner · about 2 hours

12. 12. RAG over the metrics glossary

    Which definition of active user did the bot cite, and was it the right one?

    Level 2: Intermediate · about 2 hours

13. 13. An analytics agent loop from scratch

    What does an agent actually do between your question and its answer?

    Level 3: Advanced · about 2 hours

14. 14. Audit an agent's work

    The agent says revenue grew 12%. Which step should we check, and was it right?

    Level 3: Advanced · about 2 hours

15. 15. MCP for analysts

    How does an assistant get safe, read-only access to your data?

    Level 3: Advanced · about 2 hours

By the end of the phase you will have built and tested each of these yourself, one per lesson:

• Grade an LLM classifier: A confusion matrix from two label columns, as a dict of dicts.
• Check LLM-written SQL: A guard that refuses anything but one read-only SELECT or WITH statement.
• Catch made-up numbers: A regex that pulls every number out of LLM prose: 156,142, 650.59, -0.5%, $77.9k, $1.5M, while skipping H1, Q2, FY26, 3rd and the year 2026.
• How LLMs work, for analysts: toy_tokenize: a toy tokenizer. Common words are one token, other words are cut into 4-letter pieces, and each digit and punctuation mark is a token. Real tokenizers are learned from data and cut text differently, so use this to understand the idea and the provider's own token counter for real numbers.
• Prompt versions as experiments: pair_runs: join the gold labels and two runs on ticket_id, and refuse when they are not on exactly the same tickets.
• Structured output you can load: repair: remove a code fence, cut surrounding sentences, and drop a trailing comma, without touching any value.
• How big must an eval set be?: wilson_interval and margin, then n_for_margin: the smallest eval set for a chosen plus or minus.
• Human in the loop: confidence thresholds: at_threshold: auto-handled count, right count, precision and coverage at one threshold.
• LLM as judge, and judging the judge: cohen_kappa: observed agreement, chance agreement and kappa for any two lists of grades.
• Natural language to SQL, safely: count_statements: how many statements a piece of SQL holds, without being fooled by ; inside quotes or comments.
• Keep personal data out of prompts: IN_PHONE and US_PHONE: patterns for Indian mobile and US numbers in the formats customers type.
• RAG over the metrics glossary: build_index and bm25_score: a BM25 search over the glossary, in plain Python.
• An analytics agent loop from scratch: parse_action: read one model reply and decide whether it is a tool call, an answer, or an error to report back.
• Audit an agent's work: supports and trace_lineage: link every number in the answer to the step that produced it, through any calculation, and list numbers that came from nowhere.
• MCP for analysts: validate_args: check a tool call's arguments against the tool's input schema.

Clone the course and run the first lesson's test. It fails until your exercise is right. You need only Python 3.9 or later.

git clone https://github.com/Ani19arya/analytics-from-scratch
cd analytics-from-scratch/phases/06-ai-for-analysts/01-grade-an-llm-classifier
python3 test_lesson.py

Open roles on Real Analyst Jobs where this phase's skills come up, read daily from company career pages:

• AI analyst jobs