Each case is a stakeholder's question, the SQL that gives the right answer, and how close counts. The test script reruns that SQL every time it grades, compares the analyst's number, and scores pass or fail. A person checks the answer before the case counts, because if the AI wrote the answer key, the AI is grading itself.
One gold case, in full
This case is from our NovaMart suite. NovaMart is the synthetic online store we use in the course, loaded into a Snowflake warehouse. The file is YAML, kept in git, one entry per case. Line breaks are added, the note is shortened, and two bookkeeping fields (approved_query, last_verified) are left out.
- question: >
What was our net revenue after
discounts and returns in 2024?
sql: >
select
sum(iff(status='completed',
total_amount, 0))
- sum(iff(status='returned',
total_amount, 0))
from orders
where year(order_date) = 2024
tables: [orders]
difficulty: hard
type: definitional
split: train
rel_tol: 0.002
status: verified
confidence: A
verified_by: shane
verified_at: 2026-07-08
note: >
total_amount is already net of the
discount. Correct = completed minus
returned on total_amount as-is,
2024 window.The question is the stakeholder's words, including the part that misleads. "After discounts" invites the analyst to subtract the discount. On this table that is wrong, because total_amount is already the subtotal minus the discount, plus the delivery fee.
The SQL is the definition in the note, written as a query. No number is stored, so the gold cannot drift away from the data.
When we ran this case on Jul 8, 2026, Claude Opus did not fall for the discount trap. It failed a different way: it used the neighboring total revenue entry in its dictionary, which counts completed orders and ignores returns, and came out 6.33% high. With a metric contract for net revenue, it got the gold answer three runs out of three.
Why this case uses a 0.2% tolerance
The analyst gets the question in a fresh session and nothing else, and answers with a number. Our default tolerance is 0.5% of the gold. At the 0.5% default, the reading that uses subtotal would pass. So this case sets 0.2%, and every reasonable wrong reading we could compute lands outside it:
| Wrong reading | Off by |
|---|---|
| Uses subtotal instead of total_amount | 0.40% |
| Counts completed orders only and ignores returns | 6.33% |
| Sums line totals from order_items instead of the order total | 7.39% |
| Subtracts the discount again, on both sides | 7.79% |
| Subtracts the discount again, on completed orders only | 8.27% |
| Counts cancelled orders as returns | 12.08% |
| Keeps every order status | 24.75% |
| Joins to order_items and sums the order total once per line | 87.76% |
One known hole is written into the case file. An analyst that ignores the 2024 filter comes out 0.031% away and passes, because the data runs from Jan 2, 2024 to Jan 1, 2025. The case tests the discount trap, so we accepted it.
Who signs off a gold case
Every case starts as proposed, in a candidates file, and counts toward no score. A script runs the reference SQL against the live warehouse and reports the number or the error. Then a person checks the result against the definition and signs. On this case that was Shane, on Jul 8, 2026. The verified_by field holds a real name and is never filled in by the model.
The test script can also store a fingerprint of each table's column names and types, so a renamed column marks the gold as stale instead of failing the analyst. Answer keys go stale in quieter ways too. In a run on Jun 26, 2026, a correct note about a broken purchase flag made one of our cases fail, because its answer had been computed from that flag. The context engineering page has both queries.
A gold written as SQL has one weakness. If the reference query and the analyst's query make the same mistake, the case passes. So some cases carry a number signed off from another source, like a finance report, pinned to a date. It shares no logic with the analyst's query.
How big the set should be
Start with five questions your team asked last month. Work out each answer with the metric definitions in hand, then ask the analyst. The misses tell you what context is missing.
The rule of thumb we teach is 50 to 75 cases per kind of question, taken from real requests. By that rule our 48-case NovaMart suite, the one we teach with, is small. We tune against 25 of its cases and report the score on the other 23, because nothing was tuned to those. A suite that passes everything needs harder questions. When failures cluster into a few repeating mistakes, the set is big enough to act on.
Rerun the set after every change to the analyst. How to check an AI data analyst's answer covers the checks on a single answer, and why AI gives different answers covers why a rerun can move.
In the courses
In Agentic Analytics: Build an AI Analyst, week 3 (AI evals for AI analytics), you build a golden set on NovaMart and score the analyst against it, with an AI judge for what a number cannot check. Week 4 runs the improvement loop against the same set. The free AI evals course covers the same ideas for AI features in general.