Data Engineering · Episode 02 · 1:53
Apache Iceberg explained: how to undo a bad write in your data lake
A real Iceberg table keeps every commit as a snapshot. A bad overwrite left 3 rows ($545); time travel showed the original 8 rows ($2,050), and one rollback brought them back.
What you’ll needPython 3.10+, PyArrow, PyIceberg
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. - PyArrow 18+Writing Parquet files
pip install "pyarrow>=18" - PyIceberg 0.12+Apache Iceberg tables: snapshots, time travel, rollback
pip install "pyiceberg[pyarrow,sql-sqlite]>=0.12"
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Install the packagesrequirements.txt installs every lesson's packages. This one needs only: PyArrow, PyIceberg.
pip install -r requirements.txt # or just this lesson: pip install "pyarrow>=18" "pyiceberg[pyarrow,sql-sqlite]>=0.12"
- 4Run it
cd data-engineering/16-apache-iceberg python3 src/iceberg.py
Project structure16-apache-iceberg/
Project structure
data-engineering/16-apache-iceberg/Open on GitHub →
- data/
- orders.csvtonight's 8 orders for Oct 7 ($2,050)
- short/
- lake.pya fresh SQLite catalog + warehouse folder, the orders, and total()
- undo.pythe 9-line version from the short
- src/
- iceberg.pymainthe code from the video
- lake.pya fresh SQLite catalog + warehouse folder, the orders, and total()
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/16-apache-iceberg/src/iceberg.pyOpen on GitHub →
from lake import catalog, orders, total
cat = catalog() # SQLite catalog + local folder
t = cat.create_table("shop.orders",
schema=orders.schema)
t.append(orders) # tonight's 8 orders
bad = orders.slice(0, 3) # a buggy job lost rows
t.overwrite(bad) # replaces the whole table
for n, s in enumerate(t.snapshots(), 1):
op = s.summary.operation.value
print(f"snap {n} {op}:",
s.summary["total-records"], "rows")
good = t.snapshots()[0].snapshot_id
print("now:", total(t.scan()))
print("snap 1:", total(t.scan(snapshot_id=good)))
t.manage_snapshots().rollback_to_snapshot(
good).commit()
print("rolled back:", total(t.scan()))Transcript
A folder of Parquet files can't undo a mistake. This table can. Apache Iceberg is an open table format. It gives files a history.
Think of it like Git for a table. Every write is a commit, called a snapshot. The rows stay in plain Parquet files. The metadata lists which files each snapshot uses.
A catalog points to the current metadata. A commit just swaps that pointer, all at once. In code: a catalog backed by SQLite, a warehouse folder on disk, and an orders table. Append tonight's eight orders.
That's snapshot one. Then a buggy job keeps only three rows, and overwrites the whole table. Now list the snapshots. Each records what the write did, and the rows it left.
Grab snapshot one, and read the table as it was. That's time travel. Then roll back. The current pointer moves back to snapshot one.
Run it. Three snapshots: the append of eight rows, then the overwrite, as a delete and an append of three. Right now, the table holds three rows. $545.
Snapshot one still reads eight rows. $2,050. Roll back, and the table is whole again. Nothing was deleted on disk.
The old Parquet file never moved. Only the pointer did. And because a commit swaps one pointer, readers never see a half-written table. The same trick lets you add or rename a column without rewriting a single file.
Spark, Trino, Flink and Snowflake can all work with the same Iceberg tables. One gotcha: old snapshots keep old files around, so teams expire them on a schedule. Files can't undo. Snapshots can.
That's Apache Iceberg.
Read the lesson on GitHub →- #apacheiceberg
- #dataengineering
- #datalake
- #python
