Course Wiki

Module 4: Evaluation — LLM Zoomcamp

Lessons, materials, and key concepts for module 4 of LLM Zoomcamp.

LLM Zoomcamp > Module 4: Evaluation

Module 4: Evaluation

This module covers systematic evaluation for search, RAG, and agent systems.

We generate ground truth data with an LLM. Then we measure performance with Hit Rate, MRR, and LLM-as-a-judge.

• Code notebooks are in code.
• Data is in data

Part 1: Search Evaluation

Part 1 creates a ground truth dataset and uses it to evaluate retrieval quality.

1. Intro - Why evaluation matters, offline vs online
2. Generating Ground Truth - Structured output for one document
3. Generating Ground Truth for All Documents - Batch generation, cost, and prepared data
4. Search Evaluatio

Lessons

Key concepts

• Function Calling
• RAG
• Keyword Search
• Vector Search
• Prompt Engineering
• Model Deployment
• LLM Evaluation
• OpenAI API
• Experiment Tracking
• Jupyter Notebooks
• Pandas
• Hybrid Search
• Classification Metrics
• Trading Strategy
• Streamlit
• LLM Monitoring
• Model Monitoring
• Elasticsearch

Homework and deadlines live in the course repository: llm-zoomcamp.

Other modules in this course

• Module 1: Agentic RAG
• Module 2: Vector Search
• Module 3: Orchestration
• Module 5: Monitoring
• Module 6: Best Practices
• Module 7: End-to-End Project