Data Engineering · Episode 06 · 30-second cut · 0:36
Bronze, silver, gold: why data platforms use three layers
Keep raw data in bronze and a silver bug becomes fixable: one replay took gold revenue to $1,200,704.
Key moments
What you’ll needPython 3.10+, DuckDB
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. 
DuckDB 1.0+SQL on files and tables, no database serverpip install "duckdb>=1.0"
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Install the packagesrequirements.txt installs every lesson's packages. This one needs only: DuckDB.
pip install -r requirements.txt # or just this lesson: pip install "duckdb>=1.0"
- 4Run it
cd data-engineering/20-medallion-architecture python3 scripts/make_events.py cd src python3 medallion.py
Project structure20-medallion-architecture/
Project structure
data-engineering/20-medallion-architecture/Open on GitHub →
- scripts/
- make_events.pymainwrites events.csv into src/ and short/ (seeded, 10,000 rows)
- short/
- layers.pyprints row counts per layer and the gold table
- medallion.pyrunthe code from the video
- replay.pythe 8-line version from the short
- src/
- layers.pyprints row counts per layer and the gold table
- medallion.pyrunthe code from the video
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/20-medallion-architecture/short/replay.pyOpen on GitHub →
import duckdb
from medallion import bronze, silver, gold
from layers import report
db = duckdb.connect()
bronze(db) # raw events, kept untouched
silver(db, "amount"); gold(db); report(db, "v1")
silver(db, "ltrim(amount, '$')"); gold(db)
report(db, "v2") # the fix, replayed from bronzeTranscript
Every serious data platform has three layers. Here's why. Bronze keeps raw data. Silver cleans it.
Gold answers the business. 10,000 messy order events land in bronze, untouched. Version one cleans into silver and gold, but can't read dollar amounts. Fix one expression, and replay from the same bronze.
Version one dropped 3,713 rows. Revenue, 804 thousand. Replayed: only 580 dropped, and revenue is 1.2 million.
Land it raw, clean it once, and you can always replay.
Read the lesson on GitHub →- #dataengineering
- #medallionarchitecture
- #duckdb
- #datalakehouse
