Data Engineering · Episode 03 · 30-second cut · 0:39
Your nightly diff misses changes. CDC catches every one
Same day, 6 changes: the snapshot diff saw 3 of 6, the change log caught 6 of 6.
Key moments
What you’ll needPython 3.10+
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. - No packages needed: it uses only the Python standard library.
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Run it
cd data-engineering/17-change-data-capture python3 src/cdc.py
Project structure17-change-data-capture/
Project structure
data-engineering/17-change-data-capture/Open on GitHub →
- data/
- day.sqlthe day's six writes, in order
- orders.csvthe table before the day starts: orders 1 and 2, both 'new'
- short/
- cdc.pyrunthe code from the video
- shop.pybuilds the source and target databases, loads data/
- src/
- cdc.pymainthe code from the video
- shop.pybuilds the source and target databases, loads data/
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/17-change-data-capture/short/cdc.pyOpen on GitHub →
from shop import source, triggers, snap, diff, DAY
src = source() # orders 1 and 2: 'new'
triggers(src) # every write -> changes
yday = snap(src) # last night's snapshot
src.executescript(DAY) # six writes today
seen = diff(yday, snap(src))
log = list(src.execute("SELECT * FROM changes"))
print(f"nightly diff: {len(seen)} of {len(log)}"
f" | cdc: {len(log)} of {len(log)}")Transcript
Your nightly job missed 3 updates. CDC catches every one. Change data capture logs every write, the moment it happens. Our orders table gets six writes today.
A trigger logs each write to a changes table, standing in for the log Debezium reads. The nightly job diffs last night's snapshot against tonight's. CDC just reads the log. Run it.
The diff saw three of six. CDC caught all six. A snapshot shows where you ended. The log shows how you got there.
Read the lesson on GitHub →- #changedatacapture
- #debezium
- #dataengineering
- #sql
