Course Wiki

Cross-Validation — Machine Learning Zoomcamp Module 4

Cross-validations refers to evaluating the same model on different subsets of a dataset, getting the average prediction, and spread within predictions. This method is applied in the parameter tuning step, which is the process of selecting...

Machine Learning Zoomcamp › Module 4: Evaluation Metrics for Classification › Cross-Validation

Notes

Cross-validations refers to evaluating the same model on different subsets of a dataset, getting the average prediction, and spread within predictions. This method is applied in the parameter tuning step, which is the process of selecting the best parameter.

In this algorithm, the full training dataset is divided into k partitions, we train the model in k-1 partitions of this dataset and evaluate it on the remaining subset. Then, we end up evaluating the model in all the k folds, and we calculate the average evaluation metric for all the folds.

In general, if the dataset is large, we should use the hold-out validation dataset strategy. In the other hand, if the dataset is small or we want to know the standard deviation of the model across different folds, we can use the cross-validation approach.

Libraries, classes and methods:

• Kfold(k, s, x) - sklearn.model_selection class for calculating the cross validation with k folds, s boolean attribute for shuffle decision, and an x random state
• Kfold.split(x) - sklearn.Kfold method for splitting the x dataset with the attributes established in the Kfold’s object construction.
• for i in tqdm() - library for showing the progress of each i iteration in a for loop.

The code of this project is available in this jupyter notebook.

Add notes from the video (PRs are welcome)

• Notes from Peter Ernicke

Key concepts

• Trading Strategy
• scikit-learn
• LLM Evaluation
• Jupyter Notebooks

Related notes

• mlz-m01-crisp-dm
• mlz-m01-introduction-to-numpy
• mlz-m01-introduction-to-pandas
• mlz-m01-linear-algebra-refresher
• mlz-m01-setting-up-the-environment
• mlz-m01-summary
• mlz-m02-baseline-model-for-car-price-prediction-project
• mlz-m02-categorical-variables
• mlz-m02-computing-rmse-on-validation-data
• mlz-m02-data-preparation
• mlz-m02-exploratory-data-analysis
• mlz-m02-explore-more
• mlz-m02-feature-engineering
• mlz-m02-linear-regression
• mlz-m02-linear-regression-vector-form
• mlz-m02-regularization
• mlz-m02-root-mean-squared-error-rmse
• mlz-m02-setting-up-the-validation-framework
• mlz-m02-training-linear-regression-normal-equation
• mlz-m02-tuning-the-model
• mlz-m02-using-the-model
• mlz-m03-churn-prediction-project
• mlz-m03-data-preparation
• mlz-m03-eda
• mlz-m03-explore-more
• mlz-m03-feature-importance-churn-rate-and-risk-ratio
• mlz-m03-feature-importance-correlation
• mlz-m03-feature-importance-mutual-information
• mlz-m03-logistic-regression
• mlz-m03-model-interpretation
• mlz-m03-one-hot-encoding
• mlz-m03-setting-up-the-validation-framework
• mlz-m03-summary
• mlz-m03-training-logistic-regression-with-scikit-learn
• mlz-m04-accuracy-and-dummy-model
• mlz-m04-confusion-table
• mlz-m04-evaluation-metrics-session-overview
• mlz-m04-explore-more
• mlz-m04-precision-and-recall
• mlz-m04-roc-auc
• mlz-m04-roc-curves
• mlz-m04-summary
• mlz-m05-python-virtual-environment-pipenv
• mlz-m06-decision-tree-learning-algorithm
• mlz-m06-decision-trees
• mlz-m06-ensemble-learning-and-random-forest
• mlz-m06-gradient-boosting-and-xgboost
• mlz-m06-xgboost-parameter-tuning
• mlz-m08-installation-of-tensorflow
• mlz-m09-explore-more
• mlz-m10-creating-a-pre-processing-service

Sources

• Video
• Lesson file