Data Engineering · Episode 07 · 1:51
Data contracts in Python: one renamed column, zero revenue
A renamed column made a real day of orders show $0.00. A Pydantic contract rejects that batch before load with clear errors, and Oct 7 stays $412.40.
What you’ll needPython 3.10+, DuckDB, Pydantic
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. 
DuckDB 1.0+SQL on files and tables, no database serverpip install "duckdb>=1.0"- Pydantic 2+Checks each record against a data contract
pip install "pydantic>=2"
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Install the packagesrequirements.txt installs every lesson's packages. This one needs only: DuckDB, Pydantic.
pip install -r requirements.txt # or just this lesson: pip install "duckdb>=1.0" "pydantic>=2"
- 4Run it
cd data-engineering/21-data-contracts cd src python3 contract.py
Project structure21-data-contracts/
Project structure
data-engineering/21-data-contracts/Open on GitHub →
- short/
- contract.pymainthe code from the video
- oct7.jsonOctober 7: 5 orders in the agreed shape
- oct8.jsonOctober 8: 5 orders, amount renamed to amount_usd, order_id sent as text
- shop.pythe consumer side: DuckDB warehouse, a loader with no checks, the revenue report
- src/
- contract.pyrunthe code from the video
- oct7.jsonOctober 7: 5 orders in the agreed shape
- oct8.jsonOctober 8: 5 orders, amount renamed to amount_usd, order_id sent as text
- shop.pythe consumer side: DuckDB warehouse, a loader with no checks, the revenue report
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/21-data-contracts/src/contract.pyOpen on GitHub →
import json
from typing import Literal
from pydantic import BaseModel, StrictInt
from pydantic import ValidationError
from shop import warehouse, load, report
class Order(BaseModel): # the contract
order_id: StrictInt
amount: float # required, never null
status: Literal["paid", "refunded"]
oct8 = json.load(open("oct8.json"))
db = warehouse() # Oct 7 already loaded
load(db, oct8) # no contract
report(db, "no contract:")
db = warehouse()
try:
for row in oct8:
Order.model_validate(row)
load(db, oct8) # only if every row passes
except ValidationError as e:
print("Oct 8 rejected before load:")First 24 of 28 lines
Transcript
Upstream renamed one column. Your dashboard died silently. No error, no alert. Just zero revenue for real orders.
The fix is a data contract. Like a plug and socket: agree on the shape, and the wrong plug won't fit. A contract names the columns, types, what can't be null, and allowed values. In code: Pydantic for the contract, plus a helper for our DuckDB warehouse.
order_id must be a real integer, so text fails. amount is a float with no default: required, never null. And status only allows paid or refunded. Now the producer's October 8 batch.
They renamed amount to amount_usd, and sent order_id as text. First, no contract. October 7 is already loaded, then we load October 8. The loader takes the columns it expects.
A missing one becomes null. Then the revenue report. Second run: validate every row first, and load only if all of them pass. Any failure prints each broken field, and nothing loads.
Let's run it. No contract: October 7, $412.40. October 8: five orders, zero dollars.
The orders arrived. The revenue didn't. And nothing failed. With the contract, October 8 is rejected before load.
order_id isn't a valid integer, and amount is missing. The dashboard keeps $412.40. No fake zero.
In production, both teams agree on the contract, and version it. Checks run before every load, often in the producer's CI too. Bad batches go to quarantine, with an alert naming the field. A rename becomes a new version.
Not a surprise. One renamed column can zero a dashboard. A contract makes it fail loudly, before load.
Read the lesson on GitHub →- #datacontracts
- #pydantic
- #dataengineering
- #python
