Course Wiki

Module 5: Monitoring — LLM Zoomcamp

Lessons, materials, and key concepts for module 5 of LLM Zoomcamp.

LLM Zoomcamp > Module 5: Monitoring

Module 5: Monitoring

Offline evaluation can’t tell you how your RAG system performs once real people use it. This module covers online monitoring: collecting metrics from real traffic and visualizing them on a dashboard.

We build a Streamlit chat app, capture metrics, store conversations in PostgreSQL, and create Grafana dashboards for real-time monitoring.

Lessons

Work through them in order:

1. Intro - Why monitoring matters, what we’ll build
2. Assistant Setup - Setting up the RAG assistant
3. Chat App - Basic Streamlit app with RAG
4. Capturing Metrics - LLMCallRecord, cost tracki

Lessons

Key concepts

• RAG
• PostgreSQL
• Prompt Engineering
• LLM Monitoring
• Model Monitoring
• Streamlit
• Prometheus and Grafana
• LLM Evaluation
• OpenAI API
• Docker
• Jupyter Notebooks
• Pandas
• Evidently
• Vector Search
• Kafka
• OpenTelemetry
• Elasticsearch

Homework and deadlines live in the course repository: llm-zoomcamp.

Other modules in this course

• Module 1: Agentic RAG
• Module 2: Vector Search
• Module 3: Orchestration
• Module 4: Evaluation
• Module 6: Best Practices
• Module 7: End-to-End Project