Data Engineering · Episode 11 · 30-second cut · 0:35
RAG ingestion: re-embed only what changed
Run one embeds 6 chunks, run two 0, run three only the 2 that changed, and the answer becomes 14 days.
Key moments
What you’ll needPython 3.10+, PyArrow, fastembed, LanceDB
What you’ll need
- python.org/downloads →
Python 3.10 or newerRuns the code. Check yours with python3 --version. - code.visualstudio.com →
VS Code or CursorEither works; so does any editor you like. - git-scm.com/downloads →
gitGets the code from GitHub. - PyArrow 18+Writing Parquet files
pip install "pyarrow>=18" 
fastembed 0.4+Turns text into embeddingspip install "fastembed>=0.4"
LanceDB 0.10+Stores and searches the vectorspip install "lancedb>=0.10"
Set it up
- 1Get the code
git clone https://github.com/DayanEbrar0X/data-anatomy.ai.git cd data-anatomy.ai
- 2Make a virtual environmentIt keeps this project's packages separate from the rest of your computer.
python3 -m venv .venv source .venv/bin/activate
- 3Install the packagesrequirements.txt installs every lesson's packages. This one needs only: PyArrow, fastembed, LanceDB.
pip install -r requirements.txt # or just this lesson: pip install "pyarrow>=18" "fastembed>=0.4" "lancedb>=0.10"
- 4Run it
cd data-engineering/25-data-engineering-for-rag python3 src/ingest.py
Project structure25-data-engineering-for-rag/
Project structure
data-engineering/25-data-engineering-for-rag/Open on GitHub →
- data/
- docs/
- day1/the help center on September 1: refunds.md, shipping.md, warranty.md
- refunds.md
- shipping.md
- warranty.md
- day2/the export on October 8: same files, refunds.md now says 14 days
- refunds.md
- shipping.md
- warranty.md
- day1/the help center on September 1: refunds.md, shipping.md, warranty.md
- docs/
- short/
- kb.pyloads a day's docs, creates an empty LanceDB table
- pipeline.pythe same steps as ingest.py, split into functions
- rag.pythe 9-line version from the short
- src/
- ingest.pymainthe code from the video
- kb.pyloads a day's docs, creates an empty LanceDB table
- README.mdthe idea, how to run it, a line-by-line walkthrough, things to try
data-engineering/25-data-engineering-for-rag/short/rag.pyOpen on GitHub →
from pipeline import load, stored, embed
from pipeline import upsert, ask
for day in ["day1", "day1", "day2"]:
seen = stored() # source -> hash, in LanceDB
docs = [d for d in load(day)
if seen.get(d["source"]) != d["hash"]]
rows = upsert(embed(docs)) # key: chunk id
print(f"{day}: {len(rows)} chunks embedded")
ask("How long is the refund window?")Transcript
Your chatbot is only as good as the pipeline feeding it. So production teams hash every document, every night. Read the hashes already stored in LanceDB. Same hash?
Skip it. No embedding cost. Changed? Embed it, and upsert on the chunk id.
Reruns never duplicate. Run day one, a rerun, then day two, where the refund window changed. Six chunks, then zero, then only two. And the answer: 14 days, updated today.
Embed only what changed. That keeps your chatbot fresh.
Read the lesson on GitHub →- #rag
- #dataengineering
- #lancedb
- #embeddings
