Data Engineering · Episode 08 · 30-second cut · 0:34
5,500 tiny files made this query 20x slower
Compacting 5,500 tiny Parquet files into 4 made the same query run over 20x faster.
Key moments
What you’ll needPython 3.10+, PyArrow, DuckDB
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. - PyArrow 18+Writing Parquet files
pip install "pyarrow>=18" 
DuckDB 1.0+SQL on files and tables, no database serverpip install "duckdb>=1.0"
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Install the packagesrequirements.txt installs every lesson's packages. This one needs only: PyArrow, DuckDB.
pip install -r requirements.txt # or just this lesson: pip install "pyarrow>=18" "duckdb>=1.0"
- 4Run it
cd data-engineering/22-small-files-problem cd src python3 smallfiles.py
Project structure22-small-files-problem/
Project structure
data-engineering/22-small-files-problem/Open on GitHub →
- short/
- lake.pythe click events, folder reset, the timed DuckDB query, rounded reporting
- smallfiles.pymainthe code from the video
- src/
- lake.pythe click events, folder reset, the timed DuckDB query, rounded reporting
- smallfiles.pyrunthe code from the video
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/22-small-files-problem/short/smallfiles.pyOpen on GitHub →
import pyarrow.parquet as pq
from lake import day, query, faster # 4.4M rows
def write(d, n): # same rows, split into n files
size = len(day) // n
for i in range(n):
pq.write_table(day.slice(i * size, size),
f"{d}/{i}.parquet")
write("tiny", 5500); write("big", 4)
faster(query("tiny"), query("big"))Transcript
Thousands of tiny files just made this query crawl. Same rows, same query. Only the file count changed. 4.
4 million events, and a helper that splits them into n Parquet files. Write them as 5,500 tiny files, then as four big ones. Then time the same query on both. Tiny: about half a second, and 3.
9 MB of it is footers. Big: under 50 milliseconds. Over 20 times faster. Fewer, bigger files.
Same data, faster queries.
Read the lesson on GitHub →- #parquet
- #duckdb
- #dataengineering
- #datalake
