Data Engineering · Episode 02 · 30-second cut · 0:34
Undo a bad write with Apache Iceberg
A bad overwrite left 3 rows and $545. One rollback to the earlier snapshot restored 8 rows and $2,050.
Key moments
What you’ll needPython 3.10+, PyArrow, PyIceberg
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. - PyArrow 18+Writing Parquet files
pip install "pyarrow>=18" - PyIceberg 0.12+Apache Iceberg tables: snapshots, time travel, rollback
pip install "pyiceberg[pyarrow,sql-sqlite]>=0.12"
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Install the packagesrequirements.txt installs every lesson's packages. This one needs only: PyArrow, PyIceberg.
pip install -r requirements.txt # or just this lesson: pip install "pyarrow>=18" "pyiceberg[pyarrow,sql-sqlite]>=0.12"
- 4Run it
cd data-engineering/16-apache-iceberg python3 src/iceberg.py
Project structure16-apache-iceberg/
Project structure
data-engineering/16-apache-iceberg/Open on GitHub →
- data/
- orders.csvtonight's 8 orders for Oct 7 ($2,050)
- short/
- lake.pya fresh SQLite catalog + warehouse folder, the orders, and total()
- undo.pythe 9-line version from the short
- src/
- iceberg.pymainthe code from the video
- lake.pya fresh SQLite catalog + warehouse folder, the orders, and total()
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/16-apache-iceberg/short/undo.pyOpen on GitHub →
from lake import table, orders, total
t = table() # empty Iceberg table, local disk
t.append(orders) # snapshot 1: 8 orders
t.overwrite(orders.slice(0, 3)) # buggy job
print("now:", total(t.scan()))
good = t.snapshots()[0].snapshot_id
t.manage_snapshots().rollback_to_snapshot(
good).commit()
print("rolled back:", total(t.scan()))Transcript
A folder of Parquet files can't undo a mistake. This table can. Apache Iceberg saves every write as a snapshot, and a pointer marks the current one. Make a local table, and append eight orders.
That's snapshot one. A buggy job overwrites it with three rows, and we print what's left. Undo it: roll back to snapshot one. Only the pointer moves.
Before: three rows, $545. After: all eight, $2,050. Files can't undo. Snapshots can.
Read the lesson on GitHub →- #apacheiceberg
- #dataengineering
- #datalake
- #python
