Data Engineering · Episode 10 · 1:55
Data lake vs warehouse vs lakehouse, explained with real code
The same bad write, three ways: the lake's query fails, the warehouse rejects the bad rows, and the lakehouse (a real Iceberg table) rejects them and keeps its commit history. Both answer $74.50.
What you’ll needPython 3.10+, PyArrow, DuckDB, PyIceberg
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. - PyArrow 18+Writing Parquet files
pip install "pyarrow>=18" 
DuckDB 1.0+SQL on files and tables, no database serverpip install "duckdb>=1.0"- PyIceberg 0.12+Apache Iceberg tables: snapshots, time travel, rollback
pip install "pyiceberg[pyarrow,sql-sqlite]>=0.12"
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Install the packagesrequirements.txt installs every lesson's packages. This one needs only: PyArrow, DuckDB, PyIceberg.
pip install -r requirements.txt # or just this lesson: pip install "pyarrow>=18" "duckdb>=1.0" "pyiceberg[pyarrow,sql-sqlite]>=0.12"
- 4Run it
cd data-engineering/24-lake-vs-warehouse-vs-lakehouse cd src python3 three_ways.py
Project structure24-lake-vs-warehouse-vs-lakehouse/
Project structure
data-engineering/24-lake-vs-warehouse-vs-lakehouse/Open on GitHub →
- short/
- ask_three.pythe 9-line version from the short
- store.pythe events, the bad write, the Iceberg schema, and three small helpers
- ways.pyrunthe three store functions, without the loop
- src/
- store.pythe events, the bad write, the Iceberg schema, and three small helpers
- three_ways.pymainthe code from the video
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/24-lake-vs-warehouse-vs-lakehouse/src/three_ways.pyOpen on GitHub →
import os, json, duckdb
from pyiceberg.types import StringType
from store import DAY1, DAY2, BAD, SCHEMA
from store import fresh, catalog, append
SUM = "SELECT sum(usd) FROM "
def lake(): # files in a folder, schema on read
fresh("lake")
for i, rows in enumerate([DAY1, DAY2, BAD]):
with open(f"lake/{i}.json", "w") as f:
json.dump(rows, f) # any shape lands
n = len(os.listdir("lake"))
print(f"lake: {n} files written")
return duckdb.sql(SUM + "'lake/*.json'")
def warehouse(): # typed table, schema on write
db = duckdb.connect("shop.duckdb")
db.sql("""CREATE OR REPLACE TABLE events
(id INT, kind TEXT, usd DOUBLE)""")
for rows in (DAY1, DAY2, BAD):
try:
db.executemany("INSERT INTO events "
"VALUES ($id, $kind, $usd)", rows)
except duckdb.Error:First 24 of 51 lines
Transcript
Three names for where the data lives, and they're not the same. Data lake. Data warehouse. Lakehouse.
A lake is a storage unit: cheap, takes anything. A warehouse is a library: checked on the way in, easy to find. A lakehouse keeps the cheap storage, adds the checks, and logs every change. The test: six shop events, then one bad write, an amount typed as text.
Each way answers one question: total revenue. The lake is just a folder. Each batch is dumped as a JSON file. Nothing checks it, so the bad one lands.
The schema is only worked out when DuckDB reads the files. The warehouse is a DuckDB table with types. Schema on write. A value that doesn't fit its type fails the insert.
Then the same sum. The lakehouse is an Apache Iceberg table: Parquet files plus metadata, with a SQLite catalog. Each append is a commit, checked against the schema. Adding a column only changes metadata.
No files rewritten. And every commit is a snapshot you can read back later. Same sum, through DuckDB. One loop asks all three.
Run it. The lake wrote all three files, then the query failed. The files disagree on what usd is. The warehouse rejected the bad rows: 74 dollars 50.
The lakehouse rejected them too. Two commits, same 74.50. So which?
A lake is cheapest and holds anything, but quality is the reader's problem. A warehouse gives fast, governed SQL, but data usually lives in its own storage. A lakehouse puts table rules on open files that Spark, Trino or DuckDB can all read. That's Iceberg, Delta Lake and Hudi.
Many teams use all three. Check on read. Check on write. Or check on write, over open files.
Read the lesson on GitHub →- #datalake
- #datawarehouse
- #lakehouse
- #apacheiceberg
