Course Wiki

Summary — Machine Learning Zoomcamp Module 4

Metric: A single number that describes the performance of a model Accuracy: Fraction of correct answers; sometimes misleading Precision and recall are less misleading when we have class imbalance ROC Curve: A way to evaluate the...

Machine Learning Zoomcamp › Module 4: Evaluation Metrics for Classification › Summary

Notes

General definitions:

• Metric: A single number that describes the performance of a model
• Accuracy: Fraction of correct answers; sometimes misleading
• Precision and recall are less misleading when we have class imbalance
• ROC Curve: A way to evaluate the performance at all thresholds; okay to use with imbalance
• K-Fold CV: More reliable estimate for performance (mean + std)

In brief, this weeks was about different metrics to evaluate a binary classifier. These measures included accuracy, confusion table, precision, recall, ROC curves(TPR, FPR, random model, and ideal model), and AUROC. Also, we talked about a different way to estimate the performance of the model and make the parameter tuning with cross-validation.

The code of this project is available in this jupyter notebook.

Add notes from the video (PRs are welcome)

• Notes from Maximilien Eyengue

Key concepts

• Classification Metrics
• Jupyter Notebooks

Related notes

• mlz-m01-introduction-to-numpy
• mlz-m01-introduction-to-pandas
• mlz-m01-linear-algebra-refresher
• mlz-m01-setting-up-the-environment
• mlz-m02-baseline-model-for-car-price-prediction-project
• mlz-m02-car-price-prediction-project
• mlz-m02-categorical-variables
• mlz-m02-computing-rmse-on-validation-data
• mlz-m02-data-preparation
• mlz-m02-exploratory-data-analysis
• mlz-m02-feature-engineering
• mlz-m02-linear-regression
• mlz-m02-linear-regression-vector-form
• mlz-m02-regularization
• mlz-m02-root-mean-squared-error-rmse
• mlz-m02-setting-up-the-validation-framework
• mlz-m02-training-linear-regression-normal-equation
• mlz-m02-tuning-the-model
• mlz-m02-using-the-model
• mlz-m03-data-preparation
• mlz-m03-eda
• mlz-m03-feature-importance-churn-rate-and-risk-ratio
• mlz-m03-feature-importance-correlation
• mlz-m03-feature-importance-mutual-information
• mlz-m03-logistic-regression
• mlz-m03-model-interpretation
• mlz-m03-one-hot-encoding
• mlz-m03-setting-up-the-validation-framework
• mlz-m03-training-logistic-regression-with-scikit-learn
• mlz-m04-accuracy-and-dummy-model
• mlz-m04-confusion-table
• mlz-m04-cross-validation
• mlz-m04-evaluation-metrics-session-overview
• mlz-m04-explore-more
• mlz-m04-precision-and-recall
• mlz-m04-roc-auc
• mlz-m04-roc-curves
• mlz-m06-decision-trees-parameter-tuning
• mlz-m06-gradient-boosting-and-xgboost
• mlz-m06-selecting-the-best-model
• mlz-m06-xgboost-parameter-tuning
• mlz-m08-installation-of-tensorflow
• mlz-m10-creating-a-pre-processing-service

Sources

• Video
• Lesson file