Course Wiki

Data Ingestion — LLM Zoomcamp Module 1

--- video_url: 'https://www.youtube.com/watch?v=e0owGI2JV-s&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv' code: - label: 'persistent_rag_ingest.ipynb' path: 'code/persistent_rag_ingest.ipynb' - label: 'persinsent_rag.ipynb' path...

llmz-module-01

LLM Zoomcamp › Module 1: Agentic RAG › Data Ingestion

Notes

video_url: “https://www.youtube.com/watch?v=e0owGI2JV-s&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv” code:

label: “persistent_rag_ingest.ipynb” path: “code/persistent_rag_ingest.ipynb”
label: “persinsent_rag.ipynb” path: “code/persinsent_rag.ipynb”

# Data Ingestion

So far, our RAG pipeline loads data and builds the search index at startup. With minsearch, this is fine - our FAQ dataset is small, so indexing takes less than a second. The entire pipeline runs in one process.

This breaks down as the dataset grows. Fetching data takes time - calling APIs, parsing files, cleaning text. With millions of documents, the startup becomes slow. You don’t want to wait minutes every time your service restarts.

Minsearch is in-memory. It’s a bunch of Python dictionaries bound to the process where it’s running. When you stop the process, the data disappears, so you re-index every time you restart. That’s wasteful if indexing is slow or the data takes time to prepare.

So we separate ingestion from querying. One process writes the data to a persistent search index. Another process reads from it. The two run independently and only share the index between them.

The index survives restarts, so we ingest once and query as often as we like. This is the ingestion step from data engineering. We move data from its source into a target system the application can use.

You can use any persistent search backend for this, such as Elasticsearch, OpenSearch, or Qdrant. In this module, we use sqlitesearch, a lightweight search library backed by SQLite FTS5. It has the same API as minsearch, so it’s a drop-in replacement that happens to be persistent.

I picked SQLite because it asks nothing of you. It ships with Python, so you don’t add any dependency, and it has FTS5 (full text search) built in. If you have Python, you already have a full text search engine. Using FTS5 directly is a bit awkward, so I wrote sqlitesearch as a convenient wrapper around it.

You can read more about the history of this library here.

Install it:

uv add sqlitesearch

Ingestion notebook

Create a new notebook called sqlite-ingest.ipynb (see persistent_rag_ingest.ipynb for reference). This is the ingestion process - it fetches data and writes it to a persistent index.

First, load the data using the function from ingest.py:

from ingest import load_faq_data
documents = load_faq_data()
print(f"Loaded {len(documents)} documents")

Filter to just the LLM Zoomcamp documents:

docs_llm = [doc for doc in documents if doc["course"] == "llm-zoomcamp"]
print(f"LLM Zoomcamp: {len(docs_llm)} documents")

Now create a sqlitesearch index and add documents one by one with a small delay (to simulate slow ingestion):

import time
from sqlitesearch import TextSearchIndex
index = TextSearchIndex(
    text_fields=["question", "section", "answer"],
    keyword_fields=["course"],
    db_path="faq.db"
)
for doc in docs_llm:
    index.add(doc)
    print(f"""Added: {doc["question"][:60]}...""")
    time.sleep(0.5)
index.close()
print("Done. Index saved to faq.db")

Run this notebook. You’ll see each document being added one by one. When it’s done, there’s a faq.db file on disk with the entire index. This file persists across restarts.

Querying notebook

While the ingestion is running (or after it finishes), create another notebook (see persinsent_rag.ipynb for reference).

Connect to the same database:

from sqlitesearch import TextSearchIndex
sqlite_index = TextSearchIndex(
    text_fields=["question", "section", "answer"],
    keyword_fields=["course"],
    db_path="faq.db"
)

Check how many documents are in the index:

sqlite_index.count()

Run this cell a few times while the other notebook is still ingesting. You’ll see the number growing as ingestion progres

DataTalks.Club

Data Ingestion — LLM Zoomcamp Module 1

Related Pages

Notes

Ingestion notebook

Querying notebook

Key concepts

Sources

DataTalks.Club

Data Ingestion — LLM Zoomcamp Module 1

Related Pages

Notes

Ingestion notebook

Querying notebook

Key concepts

Related notes

Sources