Data Engineering · Episode 09 · 30-second cut · 0:38
Three data jobs, one dataset
40 clean rows, 90% accuracy, a prediction served in under 1 ms: who builds which part.
Key moments
What you’ll needPython 3.10+, pandas, DuckDB, scikit-learn, joblib
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. 
pandas 2.2+Tablespip install "pandas>=2.2"
DuckDB 1.0+SQL on files and tables, no database serverpip install "duckdb>=1.0"
scikit-learn 1.4+Model selection and evaluation toolspip install "scikit-learn>=1.4"- joblib 1.3+Saves and loads a trained model
pip install "joblib>=1.3"
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Install the packagesrequirements.txt installs every lesson's packages. This one needs only: pandas, DuckDB, scikit-learn, joblib.
pip install -r requirements.txt # or just this lesson: pip install "pandas>=2.2" "duckdb>=1.0" "scikit-learn>=1.4" "joblib>=1.3"
- 4Run it
cd data-engineering/23-data-engineer-vs-data-scientist-vs-ml-engineer cd src python3 roles.py
Project structure23-data-engineer-vs-data-scientist-vs-ml-engineer/
Project structure
data-engineering/23-data-engineer-vs-data-scientist-vs-ml-engineer/Open on GitHub →
- scripts/
- make_churn.pywrites churn.csv into src/ and short/ (seeded, 42 raw rows)
- short/
- churn.csvthe raw export: id, tenure, calls, churned
- report.pyprints one line of real output per role
- roles.pymainthe code from the video
- three_jobs.pythe 9-line version from the short
- src/
- churn.csvthe raw export: id, tenure, calls, churned
- report.pyprints one line of real output per role
- roles.pyrunthe code from the video
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/23-data-engineer-vs-data-scientist-vs-ml-engineer/short/three_jobs.pyOpen on GitHub →
import duckdb
from roles import engineer, scientist, ml_engineer
from report import report
db = duckdb.connect()
first = engineer(db) # data engineer
df = engineer(db) # scheduled rerun
m, acc, base = scientist(df) # data scientist
predict, ms = ml_engineer(m) # ML engineer
report(first, df, acc, base, predict, ms)Transcript
Same data. Three jobs. Here's who builds what. One churn file.
42 raw rows. The data engineer builds a clean table, safe to rerun. The data scientist fits a model and measures it. The ML engineer saves it, and serves a fast, safe predict.
Run it. 40 clean rows, even on rerun. 90 percent accuracy, versus 50 guessing. Bad input rejected.
A churn score of 0.85, under a millisecond. Same data. One builds the table, one finds the answer, one ships it.
Read the lesson on GitHub →- #dataengineering
- #datascience
- #mlengineering
- #python
