Course Wiki

ROC Curves — Machine Learning Zoomcamp Module 4

ROC stands for Receiver Operating Characteristic, and this idea was applied during the Second World War for evaluating the strength of radio detectors. This measure considers False Positive Rate (FPR) and True Postive Rate (TPR), which are...

Machine Learning Zoomcamp › Module 4: Evaluation Metrics for Classification › ROC Curves

Notes

ROC stands for Receiver Operating Characteristic, and this idea was applied during the Second World War for evaluating the strength of radio detectors. This measure considers False Positive Rate (FPR) and True Postive Rate (TPR), which are derived from the values of the confusion matrix.

FPR is the fraction of false positives (FP) divided by the total number of negatives (FP and TN - the first row of confusion matrix), and we want to minimize it. The formula of FPR is the following:

$FPR = \large \frac{FP}{TN + FP}$

In the other hand, TPR or Recall is the fraction of true positives (TP) divided by the total number of positives (FN and TP - second row of confusion table), and we want to maximize this metric. The formula of this measure is presented below:

$TPR =\large \frac{TP}{TP + FN}$

!TPR_FPR.png

ROC curves consider Recall and FPR under all the possible thresholds. If the threshold is 0 or 1, the FPR and Recall scores are the opposite of the threshold (1 and 0 respectively), but they have different meanings, as we explained before.

We need to compare the ROC curves against a point of reference to evaluate its performance, so the corresponding curves of random and ideal models are required. It is possible to plot the ROC curves with FPR and Recall scores vs thresholds, or FPR vs Recall.

Classes and methods:

• np.repeat([x,y], [z,w]) - returns a numpy array with a z number of x values first, and then a w number of y values.
• roc_curve(x, y) - sklearn.metrics class for calculating the false positive rates, true positive rates, and thresholds, given a target x dataset and a predicted y dataset.

The entire code of this project is available in this jupyter notebook.

Add notes from the video (PRs are welcome)

• Notes from Peter Ernicke

Key concepts

• Classification Metrics
• scikit-learn
• Jupyter Notebooks

Related notes

• mlz-m01-introduction-to-numpy
• mlz-m01-introduction-to-pandas
• mlz-m01-linear-algebra-refresher
• mlz-m01-setting-up-the-environment
• mlz-m01-summary
• mlz-m02-baseline-model-for-car-price-prediction-project
• mlz-m02-car-price-prediction-project
• mlz-m02-categorical-variables
• mlz-m02-computing-rmse-on-validation-data
• mlz-m02-data-preparation
• mlz-m02-exploratory-data-analysis
• mlz-m02-explore-more
• mlz-m02-feature-engineering
• mlz-m02-linear-regression
• mlz-m02-linear-regression-vector-form
• mlz-m02-regularization
• mlz-m02-root-mean-squared-error-rmse
• mlz-m02-setting-up-the-validation-framework
• mlz-m02-training-linear-regression-normal-equation
• mlz-m02-tuning-the-model
• mlz-m02-using-the-model
• mlz-m03-data-preparation
• mlz-m03-eda
• mlz-m03-explore-more
• mlz-m03-feature-importance-churn-rate-and-risk-ratio
• mlz-m03-feature-importance-correlation
• mlz-m03-feature-importance-mutual-information
• mlz-m03-logistic-regression
• mlz-m03-model-interpretation
• mlz-m03-one-hot-encoding
• mlz-m03-setting-up-the-validation-framework
• mlz-m03-summary
• mlz-m03-training-logistic-regression-with-scikit-learn
• mlz-m04-accuracy-and-dummy-model
• mlz-m04-confusion-table
• mlz-m04-cross-validation
• mlz-m04-evaluation-metrics-session-overview
• mlz-m04-explore-more
• mlz-m04-precision-and-recall
• mlz-m04-roc-auc
• mlz-m04-summary
• mlz-m05-python-virtual-environment-pipenv
• mlz-m06-decision-tree-learning-algorithm
• mlz-m06-decision-trees
• mlz-m06-decision-trees-parameter-tuning
• mlz-m06-ensemble-learning-and-random-forest
• mlz-m06-gradient-boosting-and-xgboost
• mlz-m06-selecting-the-best-model
• mlz-m06-xgboost-parameter-tuning
• mlz-m08-installation-of-tensorflow
• mlz-m09-explore-more
• mlz-m10-creating-a-pre-processing-service

Sources

• Video
• Lesson file