Data Engineering · Episode 01 · 30-second cut · 0:34
One rerun, twice the revenue: idempotent pipelines in 30 seconds
The naive rerun gave $4,100 in 16 rows. The idempotent load gave $2,050 in 8, and Oct 6 stayed untouched.
Key moments
What you’ll needPython 3.10+, DuckDB
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. 
DuckDB 1.0+SQL on files and tables, no database serverpip install "duckdb>=1.0"
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Install the packagesrequirements.txt installs every lesson's packages. This one needs only: DuckDB.
pip install -r requirements.txt # or just this lesson: pip install "duckdb>=1.0"
- 4Run it
cd data-engineering/15-idempotent-pipelines python3 src/load.py
Project structure15-idempotent-pipelines/
Project structure
data-engineering/15-idempotent-pipelines/Open on GitHub →
- data/
- history.csv6 orders for Oct 6, already in the warehouse ($1,730)
- orders.csvtonight's 8 orders for Oct 7 ($2,050)
- short/
- job.pythe warehouse, the two-step job and the report
- rerun.pythe 9-line version from the short
- src/
- job.pythe warehouse, the two-step job and the report
- load.pymainthe code from the video
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/15-idempotent-pipelines/short/rerun.pyOpen on GitHub →
from job import naive, crash_and_rerun
def idempotent(db, day, rows):
db.begin() # all or nothing
db.execute("DELETE FROM revenue "
"WHERE day = ?", [day])
naive(db, day, rows) # the same insert
db.commit()
crash_and_rerun(naive)
crash_and_rerun(idempotent)Transcript
Your job crashed halfway. The rerun doubled the revenue. A naive load just adds rows. Rerun it, and the same eight orders land twice.
The fix: inside one transaction, delete the whole day first, then insert it fresh, and commit. Now crash and rerun both loads, on a warehouse that already holds Oct 6. Naive: $4,100 in sixteen rows. Idempotent: $2,050 in eight.
Oct 6 untouched. Reruns happen. Make every load safe to run again.
Read the lesson on GitHub →- #dataengineering
- #duckdb
- #python
- #etl
