DataTalks.Club

Articles Slack Events Podcast Wiki Books Courses

Course Wiki

Generating Ground Truth Data — LLM Zoomcamp Module 4

--- video_url: 'https://www.youtube.com/watch?v=YScoH28cVf8&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv' --- # Generating Ground Truth Data

Related Pages

llmz-module-04

LLM Zoomcamp › Module 4: Evaluation › Generating Ground Truth Data

Notes

video_url: “https://www.youtube.com/watch?v=YScoH28cVf8&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv”

Generating Ground Truth Data

To evaluate search, we need a dataset of queries where we know which document is the correct answer. This is called ground truth (or gold standard) data.

For each query in our ground truth dataset, we know which document in the knowledge base is relevant. When we run a search, we check whether the results include the correct document.

There are several ways to get ground truth data:

We don’t have a production system yet, so we’ll use an LLM to generate questions. For each FAQ document, we ask the LLM to create 5 questions that this document would answer. Then we know that for each generated question, the source document is the correct answer.

Loading the documents

We’ll use helper files from module 01 and this module.

If you don’t have them in your notebook directory, download them:

PREFIX=https://raw.githubusercontent.com/DataTalksClub/llm-zoomcamp/main

wget ${PREFIX}/cohorts/2026/01-agentic-rag/code/ingest.py
wget ${PREFIX}/cohorts/2026/01-agentic-rag/code/rag_helper.py
wget ${PREFIX}/cohorts/2026/04-evaluation/code/evaluation_utils.py

Then load the FAQ data:

from ingest import load_faq_data
documents = load_faq_data()

We’ll generate questions only for the LLM Zoomcamp FAQ. The full FAQ dataset contains documents from multiple courses. Generating five questions for every document would take longer and cost more.

documents_llm = []
for doc in documents:
    if doc["course"] == "llm-zoomcamp":
        documents_llm.append(doc)
len(documents_llm)

We’ll use these documents from now on so let’s name them as documents

documents = documents_llm

Each document already has an id field:

doc = documents[0]
print(doc["id"])
print(doc["question"])
print(doc["answer"])

The ID becomes the label in our ground truth dataset. We generate questions from a document, so we know that this document holds the answer. Later, search evaluation checks whether search brings back the document with this ID.

This is why every record needs a stable ID. If you can’t uniquely identify a document, you can’t tell whether search retrieved the right one. When you build your own evaluation set, assign an ID to each record in your knowledge base first.

Generating questions with structured output

We use an LLM to generate questions for each document.

This is the first time we’re using structured output in the course. With structured output, we ask the LLM to return data in a specific format instead of free-form text. For example, instead of getting a paragraph that contains questions, we can ask for a Python object with a questions field.

This is useful when code will process the output. The model returns the same structure every time. We can access the generated questions directly instead of parsing text manually.

We want the output as a list of strings, so we define that structure with a Pydantic model:

from pydantic import BaseModel
class Questions(BaseModel):
    questions: list[str]

The instructions for the LLM:

data_gen_instructions = """
You emulate a student who's taking our course.
Formulate 5 questions this student might ask based on a FAQ record. The record
should contain the answer to the questions, and the questions should be complete and not too short.
If possible, use as fewer words as possible from the record.

The output should resemble how people ask questions
on the internet. Not too formal, not too short, not too long.
""".strip()

We ask the LLM to use different wording from the original document. This makes the evaluation more realistic - real users won’t phrase their ques

Key concepts

Sources

DataTalks.Club. Hosted on GitHub Pages. Built with Rustkyll. We use cookies.