DataTalks.Club

Articles Slack Events Podcast Wiki Books Courses

Course Wiki

Agent Evaluation — LLM Zoomcamp Module 4

--- video_url: 'https://www.youtube.com/watch?v=2SW86BehVdI&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv' --- # Agent Evaluation

Related Pages

llmz-module-04

LLM Zoomcamp › Module 4: Evaluation › Agent Evaluation

Notes

video_url: “https://www.youtube.com/watch?v=2SW86BehVdI&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv”

Agent Evaluation

For RAG, we used the A->Q->A’ setup:

For agents, we use the same setup. A’ comes from an agent instead of a fixed RAG pipeline.

We also save the trajectory. Here, the trajectory means only the tool calls the agent made before producing the final answer.

Loading the data

Use the same ground truth questions:

import pandas as pd
df_ground_truth = pd.read_csv("data/ground_truth-new.csv")
ground_truth = df_ground_truth.to_dict(orient="records")

Load the FAQ documents and the search index:

from ingest import load_faq_data, build_index
documents = load_faq_data()
documents_llm = []
for doc in documents:
    if doc["course"] == "llm-zoomcamp":
        documents_llm.append(doc)
documents = documents_llm
index = build_index(documents)

Create a lookup table:

doc_idx = {}
for doc in documents:
    doc_idx[doc["id"]] = doc

Running the agent

Reuse the ToyAIKit agent from module 01. It handles the agent loop and stores the full message history.

First, set up the model clients:

from dotenv import load_dotenv
from openai import OpenAI
from toyaikit.llm import OpenAIClient
load_dotenv()
openai_client = OpenAI()

Define the search tool:

def search(query: str) -> list[dict]:
    """
    Search the FAQ database for entries matching the given query.
    """
    return index.search(
        query,
        num_results=5,
        boost_dict={"question": 1.0, "answer": 2.0, "section": 0.1},
        filter_dict={"course": "llm-zoomcamp"}
    )

Create the runner:

from toyaikit.tools import Tools
from toyaikit.chat.runners import OpenAIResponsesRunner
agent_tools = Tools()
agent_tools.add_tool(search)
instructions = """
You're a course teaching assistant. Answer student questions based on
the FAQ search results. Use the search tool before answering.
""".strip()
runner = OpenAIResponsesRunner(
    tools=agent_tools,
    developer_prompt=instructions,
    llm_client=OpenAIClient(model="gpt-5.4-mini")
)

The result contains:

Run it for one ground truth question:

rec = ground_truth[0]
result = runner.loop(prompt=rec["question"])

Look at the full message history:

result.all_messages

For this lesson, the trajectory is only the tool calls. We don’t need to send the full message history to the judge.

Extract the function name and arguments:

def extract_tool_calls(messages):
    tool_calls = []

    for message in messages:
        if isinstance(message, dict):
            continue

        if message.type == "function_call":
            tool_calls.append({
                "name": message.name,
                "arguments": message.arguments,
            })

    return tool_calls

For this example:

tool_calls = extract_tool_calls(result.all_messages)
tool_calls

You should see something like this:

[
    {
        "name": "search",
        "arguments": "{\"query\":\"own pace certificate at the end self-paced course certificate\"}"
    }
]

Get the original answer:

doc_id = rec["document"]
original_doc = doc_idx[doc_id]
answer_orig = original_doc["answer"]

Save the A->Q->A’ record and the trajectory:

agent_result = {
    "question": rec["question"],
    "answer_agent": result.last_message,
    "answer_orig": answer_orig,
    "tool_calls": json.dumps(tool_calls),
    "cost": result.cost.total_cost,
    "document": doc_id,
}
agent_result

The answer_agent field is what we evaluate with the LLM judge. The tool_calls field lets the judge see how the agent got there. Wrap `tool_

Key concepts

Sources

DataTalks.Club. Hosted on GitHub Pages. Built with Rustkyll. We use cookies.