Course Wiki

Model Monitoring

Detecting when a deployed model degrades: data drift, target drift, and score health over time.

Module 5 starts from the uncomfortable fact that a deployed model decays: the world shifts away from the training data. Monitoring compares production traffic against a reference dataset (the training data or a known-good period) and flags drift in feature distributions, in prediction distributions, and — where ground truth arrives later — in actual performance metrics.

The course builds two monitoring pipelines: a web service path where Prometheus scrapes Evidently metrics and Grafana dashboards them with alerts, and a batch path where Prefect jobs dump predictions to MongoDB and Evidently reports run over them. The output is actionable: dashboards and alerts that say when to retrain, not just that something changed.

Taught in

• LLM Zoomcamp

  • Module 3: Orchestration
    • AI Agents
  • Module 4: Evaluation
    • Next Steps
  • Module 5: Monitoring
    • Monitoring
    • Chat App
    • Storing Data in PostgreSQL
    • Grafana Dashboards
    • Next Steps
  • Module 6: Best Practices
    • Best Practices for RAG
  • Module 7: End-to-End Project
    • End-to-End Project Example
    • Monitoring and Containerization
    • Summary and Closing Remarks

• AI Dev Tools Zoomcamp

  • Module 4: DevOps and Observability for AI-Built Apps
    • DevOps and Observability for AI-Built Apps

• MLOps Zoomcamp

  • Module 5: Model Monitoring
    • Intro to ML monitoring
    • Evidently Monitoring Dashboard
    • Dummy monitoring
    • Data quality monitoring
    • Debugging with test suites and reports
  • Module 07: Course Project
    • Course Project ## Related concepts

• Evidently

• Prometheus and Grafana

• Model Deployment

• Experiment Tracking

• MLOps Maturity Model