Course Wiki
Next Steps — LLM Zoomcamp Module 4
--- video_url: 'https://www.youtube.com/watch?v=TlKPBjItUw8&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv' --- # Next Steps
Related Pages
LLM Zoomcamp › Module 4: Evaluation › Next Steps
Notes
video_url: “https://www.youtube.com/watch?v=TlKPBjItUw8&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv”
Next Steps
In this module, we covered evaluation at three levels:
- Search evaluation: Hit Rate and MRR to measure retrieval quality
- RAG evaluation: LLM-as-a-judge for answer quality
- Agent evaluation: final answers plus tool-call trajectories
Evaluation is not a one-time activity. As you tune search parameters, switch models, or modify prompts, re-run evaluation. Make sure the system is getting better, not worse.
Evaluation is the most important part of building AI systems. It is also the most time-consuming. Only after evaluation can you be confident that your system works. Validate every change against your evaluation framework before going to production. This applies to prompt updates, model swaps, and agent modifications.
From synthetic data to real data
The evaluation workflow in practice:
- Start with synthetic data. Use an LLM to generate questions from your FAQ or documentation. This gives you a baseline without needing real users.
- Tune the data generation. If the metrics look suspiciously good, the synthetic questions may be too close to the source text. Adjust the generation prompt to produce more realistic questions.
- Deploy and collect real data. Once the system is in production, start collecting actual user queries and feedback.
- Label real data. Have humans label whether the retrieved documents and generated answers are correct. This produces the most reliable ground truth.
- Tune synthetic generation to match real data. Use the patterns from real queries to improve your synthetic data generator. The closer your synthetic data is to real data, the more useful the metrics become.
Nothing beats manual evaluation. Try the system yourself, think about edge cases, and collect examples of where it fails. This is especially important in the early stages when you don’t have automated evaluation set up yet.
Evaluation frameworks
For production systems, consider using evaluation frameworks that make it easier to manage test datasets, run evaluations, and track results:
- Ragas: a framework for evaluating RAG systems with metrics like faithfulness, answer relevance, and context precision
- DeepEval: provides built-in metrics for RAG evaluation including hallucination detection and answer relevance
- TruLens: instruments your LLM app and tracks quality metrics
These frameworks implement many of the concepts we covered here and add visualizations and experiment tracking.
Monitoring
Online evaluation (monitoring) is what you do after deploying your system.
Key approaches:
- User feedback: thumbs up/down buttons to collect signal
- Logging: record queries, retrieved documents, and answers
- Dashboards: track metrics over time to spot degradation
- Alerts: get notified when metrics drop below a threshold
Monitoring is covered in more detail in module 05.
To learn more
See also:
- Cohorts and materials:
- 2024 cohort evaluation module (uses Elasticsearch): 2024/04-monitoring
- 2025 cohort evaluation module: 2025/03-evaluation
Key concepts
- RAG
- Experiment Tracking
- Classification Metrics
- Prompt Engineering
- LLM Monitoring
- Model Monitoring
- Elasticsearch
- Model Deployment
- LLM Evaluation
Related notes
- llmz-m01-agents
- llmz-m01-building-the-prompt
- llmz-m01-data-ingestion
- llmz-m01-function-calling
- llmz-m01-introduction
- llmz-m01-other-frameworks
- llmz-m01-quick-rag-revision-optional
- llmz-m01-rag
- llmz-m01-rag-helper
- llmz-m01-search
- llmz-m01-the-agentic-loop
- llmz-m01-the-course-faq-dataset
- llmz-m01-the-llm
- llmz-m01-toyaikit
- llmz-m01-wrap-up-of-part-1
- llmz-m02-embeddings
- llmz-m02-next-steps
- llmz-m02-rag-with-vector-search
- llmz-m02-using-onnx-runtime-instead-of-pytorch
- llmz-m02-vector-search
- llmz-m02-vector-search-with-minsearch
- llmz-m02-vector-search-with-pgvector
- llmz-m02-vector-search-with-sqlitesearch
- llmz-m03-ai-agents
- llmz-m03-ai-copilot
- llmz-m03-ai-orchestration
- llmz-m03-best-practices
- llmz-m03-context-engineering
- llmz-m03-multi-agent-systems
- llmz-m03-next-steps
- llmz-m03-retrieval-augmented-generation
- llmz-m04-agent-evaluation
- llmz-m04-evaluation
- llmz-m04-generating-ground-truth-data
- llmz-m04-generating-ground-truth-for-all-documents
- llmz-m04-generating-rag-answers
- llmz-m04-llm-as-a-judge
- llmz-m04-rag-and-agent-evaluation
- llmz-m04-search-evaluation
- llmz-m04-search-evaluation-metrics
- llmz-m04-search-parameter-tuning
- llmz-m05-assistant
- llmz-m05-built-in-judge
- llmz-m05-capturing-metrics
- llmz-m05-chat-app
- llmz-m05-docker-compose
- llmz-m05-feedback-dashboard
- llmz-m05-grafana-dashboards
- llmz-m05-monitoring
- llmz-m05-next-steps
- llmz-m05-querying-data
- llmz-m05-storing-data-in-postgresql
- llmz-m05-streamlit-dashboard
- llmz-m05-synthetic-data-generation
- llmz-m05-user-feedback
- llmz-m06-best-practices-for-rag
- llmz-m06-document-reranking
- llmz-m06-hybrid-search-with-langchain
- llmz-m06-next-steps
- llmz-m07-chunking-for-longer-texts
- llmz-m07-content-processing-cases-and-steps
- llmz-m07-end-to-end-project-example
- llmz-m07-evaluating-rag
- llmz-m07-evaluating-retrieval
- llmz-m07-interface-and-ingestion-pipeline
- llmz-m07-monitoring-and-containerization
- llmz-m07-summary-and-closing-remarks