Data Engineering · Episode 10 · 30-second cut · 0:37
Lake vs warehouse vs lakehouse in 30 seconds
One bad write: the lake breaks, the warehouse and the lakehouse reject it and both answer $74.50.
Key moments
What you’ll needPython 3.10+, PyArrow, DuckDB, PyIceberg
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. - PyArrow 18+Writing Parquet files
pip install "pyarrow>=18" 
DuckDB 1.0+SQL on files and tables, no database serverpip install "duckdb>=1.0"- PyIceberg 0.12+Apache Iceberg tables: snapshots, time travel, rollback
pip install "pyiceberg[pyarrow,sql-sqlite]>=0.12"
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Install the packagesrequirements.txt installs every lesson's packages. This one needs only: PyArrow, DuckDB, PyIceberg.
pip install -r requirements.txt # or just this lesson: pip install "pyarrow>=18" "duckdb>=1.0" "pyiceberg[pyarrow,sql-sqlite]>=0.12"
- 4Run it
cd data-engineering/24-lake-vs-warehouse-vs-lakehouse cd src python3 three_ways.py
Project structure24-lake-vs-warehouse-vs-lakehouse/
Project structure
data-engineering/24-lake-vs-warehouse-vs-lakehouse/Open on GitHub →
- short/
- ask_three.pythe 9-line version from the short
- store.pythe events, the bad write, the Iceberg schema, and three small helpers
- ways.pyrunthe three store functions, without the loop
- src/
- store.pythe events, the bad write, the Iceberg schema, and three small helpers
- three_ways.pymainthe code from the video
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/24-lake-vs-warehouse-vs-lakehouse/short/ask_three.pyOpen on GitHub →
import duckdb
from ways import lake, warehouse, lakehouse
for way in (lake, warehouse, lakehouse):
name = way.__name__
try:
usd = way().fetchone()[0]
print(f"{name}: revenue ${usd:.2f}")
except duckdb.Error:
print(f"{name}: query failed")Transcript
Three names for where the data lives, and they're not the same. Lake, warehouse, lakehouse. Same six events, plus one bad write: an amount typed as text. One loop asks each store for total revenue.
Run it. The lake took the bad file, then its query failed. The warehouse rejected the bad rows: 74.50.
The lakehouse rejected them too, on open Parquet files, with two commits. Check on read. Check on write. Or check on write, over open files.
Read the lesson on GitHub →- #datalake
- #datawarehouse
- #lakehouse
- #apacheiceberg
