Data Engineering · Episode 12 · 1:47
Batch vs streaming explained: when data can't wait for the nightly job
Same fraud rule, two ways: streaming flags the card after 2 seconds with $1,442 spent; the nightly batch flags it after 11 h 52 m with $4,202 spent. A minimal simulation of what Kafka and Flink do in production.
What you’ll needPython 3.10+
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. - No packages needed: it uses only the Python standard library.
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Run it
cd data-engineering/26-batch-vs-streaming python3 src/stream.py
Project structure26-batch-vs-streaming/
Project structure
data-engineering/26-batch-vs-streaming/Open on GitHub →
- data/
- swipes.csv12 card swipes on 2026-10-08: time, card, country, usd
- short/
- fraud.pythe 9-line version from the short
- swipes.pyloads swipes.csv, the 02:00 batch slot, the alert report
- src/
- stream.pymainthe code from the video
- swipes.pyloads swipes.csv, the 02:00 batch slot, the alert report
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/26-batch-vs-streaming/src/stream.pyOpen on GitHub →
from collections import deque
from datetime import timedelta
from swipes import day, NIGHTLY, report
WINDOW = timedelta(seconds=60)
LAG = timedelta(seconds=2) # simulated delay
def suspicious(win, e): # 3 in 60 s, 2+ countries
w = win.setdefault(e.card, deque())
w.append(e)
while e.ts - w[0].ts > WINDOW:
w.popleft()
countries = {x.country for x in w}
return len(w) >= 3 and len(countries) >= 2
# Batch: one job at 02:00 reads the whole day
win = {}
for e in day:
if suspicious(win, e):
report("batch", e, NIGHTLY)
# Streaming: a Kafka-style topic, simulated
topic, committed, win = [], 0, {}
for e in day: # swipes arrive liveFirst 24 of 30 lines
Transcript
Some data can't wait for the nightly job. Batch handles a whole day at once. Streaming handles each event as it lands. Batch is the morning paper.
Streaming is the alert on your phone. Here's one day of card swipes: twelve of them. At 2:07 pm, card 4417 swipes in the US, then twice in the UK, inside one minute. That's our fraud rule: three swipes in 60 seconds, from two countries.
In code: a 60-second window, and a simulated two-second delivery delay. The rule keeps a small window of recent swipes per card. Swipes older than 60 seconds fall out. Three left, from two countries or more?
Suspicious. Batch: one job at 2 a.m. reads the whole day, in one pass.
Streaming, here, is a minimal simulation of a Kafka-style topic: a log where every event gets an offset. The producer appends each swipe. The consumer reads the next offset, runs the same rule, flagging two seconds after the swipe, and commits the offset, so after a crash it resumes right there. Let's run it.
Batch flags card 4417 eleven hours and 52 minutes after the swipe. By then, the card has spent 4,202 dollars. Streaming flags it two seconds after the swipe. Spent: 1,442.
Committed offset: 12. All twelve swipes processed. Same rule, same data. The only difference is when the code runs.
In production, Kafka holds the topic, and Flink runs the windows, with state that survives restarts. Batch still wins for reports and backfills: cheaper, simpler, easy to rerun. So ask how late is too late. For fraud, two seconds beats almost twelve hours.
Read the lesson on GitHub →- #kafka
- #streaming
- #dataengineering
- #frauddetection
