Model Anatomy · Episode 05 · 30-second cut · 0:37
LLM evals: from 60% to 100% without guessing
Ten real tickets, scored on every change: version 1 hits 60%, version 2 80%, version 3 100%. Keep the tickets as a regression gate and every release is measured.
Key moments
What you’ll needPython 3.10+
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. - No packages needed: it uses only the Python standard library.
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Run it
cd methodologies/05-evals-loop-engineering python3 src/evals.py
Project structure05-evals-loop-engineering/
Project structure
methodologies/05-evals-loop-engineering/Open on GitHub →
- data/
- cases.csvthe eval set: 10 tickets and the right team
- short/
- loop.pythe 7-line version from the short
- tickets.pycases, router and three rule versions
- src/
- cases.pyloads cases.csv
- evals.pymainthe code from the video
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
methodologies/05-evals-loop-engineering/short/loop.pyOpen on GitHub →
from tickets import CASES, route, VERSIONS
for rules in VERSIONS:
ok = [route(t, rules) == want
for t, want in CASES]
rate = sum(ok) / len(ok)
print(f"{rate:.0%} pass")Transcript
Your AI aced the demo, then got four out of ten tickets wrong. The fix is a loop: run, score, fix, rerun. Ten real tickets with right answers: a practice test for your AI. Each version of the router runs every ticket.
The pass rate is the share it got right. Version one: 60%. Read the failures, fix one thing, rerun. 80%.
Then 100%. Keep those tickets. Every future change must pass them too. Run, score, fix, rerun.
Read the lesson on GitHub →- #llmevals
- #aiengineering
- #llm
- #python
