Build Lab · Lab 01 · 1:44
Build a data pipeline in Python: API to Parquet in 3 files
Fetch paginated JSON, clean it with pandas, write Parquet and query it with DuckDB, a project you can build in a day. 120 of 122 rows kept, 19.0 KB of JSON down to 5.6 KB of Parquet, and 99 paid vs 21 refunded with no database server.
What you’ll needPython 3.10+, pandas, PyArrow, DuckDB
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. 
pandas 2.2+Tablespip install "pandas>=2.2"- PyArrow 15+Writing Parquet files
pip install "pyarrow>=15" 
DuckDB 1.0+SQL on files and tables, no database serverpip install "duckdb>=1.0"
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Install the packagesrequirements.txt installs every lesson's packages. This one needs only: pandas, PyArrow, DuckDB.
pip install -r requirements.txt # or just this lesson: pip install "pandas>=2.2" "pyarrow>=15" "duckdb>=1.0"
- 4Run it
cd data-engineering/build-lab-01-api-to-parquet cd orders_pipeline python3 main.py
Project structurebuild-lab-01-api-to-parquet/
Project structure
data-engineering/build-lab-01-api-to-parquet/Open on GitHub →
- orders_pipeline/
- api/recorded API responses (3 pages)
- orders_p1.json
- orders_p2.json
- orders_p3.json
- data/created when you run it
- scripts/
- make_pages.pymade the recorded API pages
- fetch.py1. fetch every page from the API
- main.pymain3. write Parquet and query it with DuckDB
- transform.py2. clean the rows with pandas
- api/recorded API responses (3 pages)
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/build-lab-01-api-to-parquet/orders_pipeline/main.pyOpen on GitHub →
import duckdb
from pathlib import Path
from fetch import fetch_orders
from transform import clean
PQ = "data/orders.parquet"
rows = fetch_orders()
df = clean(rows)
df.to_parquet(PQ, index=False)
kb = lambda p: Path(p).stat().st_size / 1024
j = sum(kb(p) for p in Path("api").glob("*"))
print(len(rows), "rows fetched,", len(df), "kept")
print(f"JSON {j:.1f} KB -> Parquet {kb(PQ):.1f} KB")
sql = f"SELECT status, count(*) FROM '{PQ}'"
sql += " GROUP BY 1 ORDER BY 1"
print(duckdb.sql(sql).fetchall())Transcript
This tiny pipeline turns messy API data into a file three times smaller, and instantly queryable. Here's a project you can build in a day. Let's build it. Fetch, clean, write.
Three files. Our source is an orders API. It sends results in pages, forty orders at a time. File one: fetch.
py. We start a list for the rows, and begin on page one. While there's a page, we read it. Here it's a recorded response, so every run is repeatable.
In production, this line is one requests.get call. We add each page's orders, and follow next page until there isn't one. File two: transform.
py. Pandas turns the rows into a table. The API sends amounts as text, so we make them numbers, and timestamps become real dates. The API repeated two orders between pages, so we drop duplicates by order ID.
File three: main.py ties it together. Fetch, clean, and write it as Parquet. Then compare file sizes, and ask DuckDB a question, straight from the file.
Let's run it. 122 rows fetched, 120 kept. 19 KB of JSON became 5.6 KB of Parquet.
Over three times smaller. And DuckDB counts 99 paid and 21 refunded, with no database server. Why Parquet? It stores each column together, typed and compressed.
So Spark, your warehouse, pandas, and DuckDB read only the columns they need. Fetch, clean, write. A real data pipeline, in three files.
Read the lesson on GitHub →- #dataengineering
- #python
- #parquet
- #duckdb
