Course Wiki

LLM Evaluation

Measuring retrieval quality and answer quality for RAG systems — before and after deployment.

Module 4 answers ‘did my RAG change actually help?’. Evaluation splits in two: retrieval metrics (did the right documents make it into the context — hit rates, MRR over a golden test set of question-document pairs) and answer quality (is the generated answer correct, relevant, and grounded — judged by an LLM-as-a-judge against the retrieved documents).

The course builds a golden dataset from real user questions, then runs offline evaluation over candidate configurations (different retrieval methods, prompts, models) before anything ships. Online evaluation then continues the measurement in production from user feedback and monitored behavior — the handoff to the monitoring module.

Taught in

• Machine Learning Zoomcamp

  • Module 1: Introduction to Machine Learning
    • CRISP-DM
    • Summary
  • Module 4: Evaluation Metrics for Classification
    • Evaluation metrics: session overview
    • Cross-Validation

• LLM Zoomcamp

  • Module 1: Agentic RAG
    • Building the Prompt
  • Module 2: Vector Search
    • Embeddings
    • Next Steps
  • Module 4: Evaluation
    • Evaluation
    • Generating Ground Truth Data
    • Generating Ground Truth for All Documents
    • Search Evaluation
    • Search Evaluation Metrics
    • Search Parameter Tuning
    • RAG and Agent Evaluation
    • Generating RAG Answers
    • LLM as a Judge
    • Agent Evaluation
    • Next Steps
  • Module 5: Monitoring
    • Monitoring
    • Capturing Metrics
    • User Feedback
    • Built-in Judge
    • Next Steps
  • Module 6: Best Practices
    • Best Practices for RAG
    • Document Reranking
  • Module 7: End-to-End Project
    • End-to-End Project Example
    • Evaluating Retrieval
    • Evaluating RAG
    • Monitoring and Containerization
    • Summary and Closing Remarks
    • Chunking for Longer Texts
    • Content Processing Cases and Steps

• MLOps Zoomcamp

  • Module 07: Course Project
    • Course Project

• Stock Markets Analytics Zoomcamp

  • Module 3: Analytical Modeling
    • Analytical Modeling ## Related concepts

• LLM Monitoring

• RAG

• Vector Search

• Classification Metrics

• Agentic RAG