Course Wiki

LLM Monitoring

Watching a live LLM application: user feedback, system health, and drift in what users ask.

Module 5 keeps watch after deployment. The signals are different from classic ML: explicit user feedback (thumbs up/down on answers), implicit signals (users rephrasing questions, abandoning conversations), system health (latency, errors, token spend), and the distribution of incoming questions drifting toward topics the knowledge base does not cover.

The course builds the monitoring loop with the dlt workshop’s trace pipelines feeding dashboards — queries, retrieved documents, generated answers, and feedback all land in DuckDB for analysis. Monitoring closes the loop that evaluation opened: what you measure in production becomes the next round of test cases and improvements.

Taught in

• LLM Zoomcamp

  • Module 4: Evaluation
    • Next Steps
  • Module 5: Monitoring
    • Monitoring
    • User Feedback
    • Feedback Dashboard
    • Grafana Dashboards
  • Module 7: End-to-End Project
    • Monitoring and Containerization ## Related concepts

• LLM Evaluation

• dlt

• RAG

• Model Monitoring