Model Anatomy · Episode 13 · 30-second cut · 0:39
How PySpark handles data too big for one machine
100 million sales go into 4 partitions with one lazy plan, and nothing runs until the action. Then store 3 wins with $479.6 million.
Key moments
What you’ll needPython 3.10+, PySpark, Java 17 or 21
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. 
PySpark 3.5+Spark, run locallypip install "pyspark>=3.5"- Java 17 or 21Spark runs on the Java Virtual Machine, so pip alone is not enough
brew install openjdk@21
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Install the packagesrequirements.txt installs every lesson's packages. This one needs only: PySpark.
pip install -r requirements.txt # or just this lesson: pip install "pyspark>=3.5"
- 4Install Java 17 or 21Spark runs on the Java Virtual Machine, so pip alone is not enough
brew install openjdk@21 # macOS sudo apt install openjdk-21-jdk # Debian / Ubuntu java -version
- 5Run it
cd data-engineering/13-pyspark python3 src/sales.py
Project structure13-pyspark/
Project structure
data-engineering/13-pyspark/Open on GitHub →
- src/
- sales.pymainthe 22 lines from the video
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
top.pyOpen on GitHub →
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = (spark.range(100_000_000, numPartitions=4)
.selectExpr("id % 50 AS store",
"abs(hash(id)) % 500 AS amt"))
print("partitions:", df.rdd.getNumPartitions())
top = (df.where("amt >= 100").groupBy("store")
.sum("amt").toDF("store", "rev"))
print(top.orderBy("rev", ascending=False).first())Transcript
Too much data for one machine? Spark splits it. Each core takes a slice, all at once. A hundred million sales, in four partitions.
Each sale gets a store and an amount. Filter, group by store, sum. Nothing runs yet. Spark only builds a plan.
The work starts at the action: first(). Each partition sums its own stores. Then the shuffle: a store's totals meet in one place. Run it: four partitions, and store 3 wins with 479.
6 million. Split. Plan. Act once.
That's how Spark scales.
Read the lesson on GitHub →- #pyspark
- #apachespark
- #dataengineering
- #bigdata
