Course Wiki

Classification Metrics

Accuracy, precision, recall, F1, confusion matrices, ROC curves and AUC — and why accuracy alone misleads on imbalanced data.

Module 4 confronts the class-imbalance problem: when 3% of customers churn, a model that predicts ‘no churn’ for everyone is 97% accurate and useless. The course builds the confusion matrix and derives precision (how many predicted churners actually churn) and recall (how many real churners were caught), then combines them into F1.

The second half covers threshold-independent evaluation: ROC curves, the area under them (AUC), and cross-validation to compare model variants reliably. The module’s outcome is an evaluation workflow that matches the metric to the business decision — for churn, that is usually a precision-recall tradeoff at a chosen threshold.

Taught in

• Machine Learning Zoomcamp

  • Module 2: Machine Learning for Regression
    • Car price prediction project
    • Root Mean Squared Error (RMSE)
    • Computing RMSE on validation data
    • Feature engineering
    • Using the model
  • Module 4: Evaluation Metrics for Classification
    • Evaluation metrics: session overview
    • Precision and Recall
    • ROC Curves
    • ROC AUC
    • Summary
    • Explore more
  • Module 6: Decision Trees and Ensemble Learning
    • Decision trees parameter tuning
    • XGBoost parameter tuning
    • Selecting the best model

• LLM Zoomcamp

  • Module 2: Vector Search
    • Vector Search with sqlitesearch
  • Module 4: Evaluation
    • Search Evaluation Metrics
    • Next Steps

• MLOps Zoomcamp

  • Module 1: Introduction
    • Homework ## Related concepts

• Logistic Regression

• Regularization

• Gradient Boosting

• LLM Evaluation

• Neural Networks