Course Wiki
Monitoring — LLM Zoomcamp Module 5
--- video_url: 'https://www.youtube.com/watch?v=lbEj3Waxs1U&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv' --- # Monitoring
Related Pages
LLM Zoomcamp › Module 5: Monitoring › Monitoring
Notes
video_url: “https://www.youtube.com/watch?v=lbEj3Waxs1U&list=PL3MmuxUbc_hLZFNgSad56pDBKK8KO0XIv”
Monitoring
In the previous module we evaluated our system offline, before anyone used it. We measured search quality with Hit Rate and MRR, and answer quality with cosine similarity and an LLM judge.
But once we deploy, real people start asking real questions. The offline numbers stop telling the whole story. We don’t know how long answers take, what they cost, or whether anyone finds them useful. We need to watch the system while it runs.
That’s monitoring: online evaluation. We collect metrics from the running system and put them on a dashboard. Then we can see how it performs with real traffic.
For every question that comes in, there’s a lot we can capture:
- The instructions, prompt, and model behind the answer
- Input and output tokens, and how much the call cost
- Response time: how long the person waited
- User feedback: a thumbs up or thumbs down on the answer
- Relevance: does the answer address the question?
Collecting and viewing all of this takes three new pieces:
- a user interface where people ask questions
- a database to store conversations and feedback
- a dashboard to visualize the metrics
We build all three on top of the RAG pipeline from the earlier modules. We don’t rebuild the RAG part. We wrap it in a Streamlit app and save every interaction to PostgreSQL. Then we put a dashboard in front of the data. At the end we add Grafana for a more powerful view.
We focus on RAG here. Monitoring an agent works almost the same way, so we leave it as homework. The agents module already has the pieces you need to apply these same ideas there.
Key concepts
- RAG
- PostgreSQL
- Prompt Engineering
- LLM Monitoring
- Model Monitoring
- Streamlit
- Prometheus and Grafana
- LLM Evaluation
Related notes
- llmz-m01-agents
- llmz-m01-building-the-prompt
- llmz-m01-data-ingestion
- llmz-m01-function-calling
- llmz-m01-introduction
- llmz-m01-other-frameworks
- llmz-m01-quick-rag-revision-optional
- llmz-m01-rag
- llmz-m01-rag-helper
- llmz-m01-search
- llmz-m01-the-agentic-loop
- llmz-m01-the-course-faq-dataset
- llmz-m01-the-llm
- llmz-m01-toyaikit
- llmz-m01-wrap-up-of-part-1
- llmz-m02-embeddings
- llmz-m02-next-steps
- llmz-m02-rag-with-vector-search
- llmz-m02-vector-search
- llmz-m02-vector-search-with-minsearch
- llmz-m02-vector-search-with-pgvector
- llmz-m02-vector-search-with-sqlitesearch
- llmz-m03-ai-agents
- llmz-m03-ai-copilot
- llmz-m03-ai-orchestration
- llmz-m03-best-practices
- llmz-m03-context-engineering
- llmz-m03-multi-agent-systems
- llmz-m03-next-steps
- llmz-m03-retrieval-augmented-generation
- llmz-m04-agent-evaluation
- llmz-m04-evaluation
- llmz-m04-generating-ground-truth-data
- llmz-m04-generating-ground-truth-for-all-documents
- llmz-m04-generating-rag-answers
- llmz-m04-llm-as-a-judge
- llmz-m04-next-steps
- llmz-m04-rag-and-agent-evaluation
- llmz-m04-search-evaluation
- llmz-m04-search-evaluation-metrics
- llmz-m04-search-parameter-tuning
- llmz-m05-assistant
- llmz-m05-built-in-judge
- llmz-m05-capturing-metrics
- llmz-m05-chat-app
- llmz-m05-docker-compose
- llmz-m05-feedback-dashboard
- llmz-m05-grafana-dashboards
- llmz-m05-next-steps
- llmz-m05-querying-data
- llmz-m05-storing-data-in-postgresql
- llmz-m05-streamlit-dashboard
- llmz-m05-synthetic-data-generation
- llmz-m05-user-feedback
- llmz-m06-best-practices-for-rag
- llmz-m06-document-reranking
- llmz-m06-next-steps
- llmz-m07-chunking-for-longer-texts
- llmz-m07-content-processing-cases-and-steps
- llmz-m07-end-to-end-project-example
- llmz-m07-evaluating-rag
- llmz-m07-evaluating-retrieval
- llmz-m07-interface-and-ingestion-pipeline
- llmz-m07-monitoring-and-containerization
- llmz-m07-summary-and-closing-remarks