Model Anatomy · Episode 11 · 30-second cut · 0:36
Why 80% accuracy can catch zero churners
If 20% of customers leave, always guessing "stays" scores 80%. Set the goal first, recall, and logistic regression catches 76% of the churners.
Key moments
What you’ll needPython 3.10+, pandas, scikit-learn
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. 
pandas 2.2+Tablespip install "pandas>=2.2"
scikit-learn 1.4+Model selection and evaluation toolspip install "scikit-learn>=1.4"
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Install the packagesrequirements.txt installs every lesson's packages. This one needs only: pandas, scikit-learn.
pip install -r requirements.txt # or just this lesson: pip install "pandas>=2.2" "scikit-learn>=1.4"
- 4Run it
cd machine-learning/11-choosing-a-model python3 src/choose.py
Project structure11-choosing-a-model/
Project structure
machine-learning/11-choosing-a-model/Open on GitHub →
- data/
- customers.csv3,000 customers: tenure, monthly_charges, support_tickets, contract, churn
- scripts/
- make_customers.pymade customers.csv (seeded, so it is the same every run)
- src/
- choose.pymainthe 22 lines from the video
- customers.pyloads customers.csv into a pandas table
- models.pythe four candidates, each behind the same preprocessing
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
pick.pyOpen on GitHub →
from sklearn.model_selection import cross_validate
from churn import load_churn, candidates
X, y = load_churn()
M, rec = ["accuracy", "recall"], {}
for n, clf in candidates().items():
r = cross_validate(clf, X, y, cv=5, scoring=M)
acc, rec[n] = (r["test_"+m].mean() for m in M)
print(f"{n:9} acc {acc:.2f} rec {rec[n]:.2f}")
print("pick:", max(rec, key=rec.get))Transcript
This model is 80% accurate, and catches no one. Say 20% of customers churn. Guess nobody leaves, and you're 80% right. The fix: load 3,000 customers, pick the goal first: recall, the share of churners we catch.
Every candidate gets the same five folds. Average the scores, and pick the best recall. The baseline: 80% accuracy, zero recall. Logistic regression: 73% accurate, but it catches 76%.
So logistic wins. Pick the goal first, then the model. Accuracy alone can lie.
Read the lesson on GitHub →- #machinelearning
- #datascience
- #mlops
- #python
