Data Engineering · Episode 12 · 30-second cut · 0:39
Batch vs streaming: 2 seconds vs 12 hours
Streaming caught the fraud after 2 s and $1,442; batch after 11 h 52 m and $4,202.
Key moments
What you’ll needPython 3.10+
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. - No packages needed: it uses only the Python standard library.
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Run it
cd data-engineering/26-batch-vs-streaming python3 src/stream.py
Project structure26-batch-vs-streaming/
Project structure
data-engineering/26-batch-vs-streaming/Open on GitHub →
- data/
- swipes.csv12 card swipes on 2026-10-08: time, card, country, usd
- short/
- fraud.pythe 9-line version from the short
- swipes.pyloads swipes.csv, the 02:00 batch slot, the alert report
- src/
- stream.pymainthe code from the video
- swipes.pyloads swipes.csv, the 02:00 batch slot, the alert report
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/26-batch-vs-streaming/short/fraud.pyOpen on GitHub →
from swipes import day, NIGHTLY, LAG
from swipes import suspicious, report
batch, stream = {}, {}
for e in day: # streaming: swipe by swipe, live
if suspicious(stream, e):
report("stream", e, e.ts + LAG)
for e in day: # batch: the whole day at 02:00
if suspicious(batch, e):
report("batch", e, NIGHTLY)Transcript
Some data can't wait for the nightly job. One fraud rule: three card swipes in 60 seconds, from two countries. Streaming, simulated here, checks every swipe the moment it lands. Batch runs the exact same check at 2 a.
m., on the whole day. Streaming flags card 4417 two seconds after the swipe. 1,442 dollars spent.
Batch flags it eleven hours and 52 minutes later. 4,202 dollars gone. Same rule. Only the timing differs.
In production: Kafka and Flink.
Read the lesson on GitHub →- #kafka
- #streaming
- #dataengineering
- #frauddetection
