Course Wiki

Ensemble learning and random forest — Machine Learning Zoomcamp Module 6

Ensemble learning is a machine learning paradigm where multiple models, often referred to as 'weak learners', are strategically combined to solve a particular computational intelligence problem. This approach frequently yields superior...

Machine Learning Zoomcamp › Module 6: Decision Trees and Ensemble Learning › Ensemble learning and random forest

Notes

Ensemble learning is a machine learning paradigm where multiple models, often referred to as ‘weak learners’, are strategically combined to solve a particular computational intelligence problem. This approach frequently yields superior predictive performance compared to using a single model.

Random Forest is an example of ensemble learning where each model is a decision tree and their predictions are aggregated to identify the most popular result. Random forest only selects a random subset of features from the original data to make predictions. The ‘randomness’ in Random Forest stems from two key aspects:

• Each tree is potentially trained on a bootstrapped sample of the original data, introducing randomness at the row level.
• At each node during tree construction, only a random subset of features is considered for splitting. This feature randomness helps decorrelate the trees, preventing overfitting and promoting generalization to unseen data.

Bootstrapping is a resampling technique where numerous subsets of the data are created by sampling the original data with replacement. This means that some data points may appear multiple times in a single bootstrap sample, while others may be excluded. In Random Forest, each decision tree is trained on a distinct bootstrap sample, further contributing to the diversity and robustness of the ensemble.

Parameter tuning is crucial for optimizing the performance of a Random Forest model. Two critical parameters are max_depth, which controls the maximum depth of each decision tree, and n_estimators, which determines the number of trees in the forest. Increasing max_depth allows for more complex trees, potentially leading to overfitting. Conversely, a larger n_estimators generally improves model accuracy but increases computational cost.

In random forests, the decision trees are trained independently to each other.

Classes, functions, and methods:

• from sklearn.ensemble import RandomForestClassifier: random forest classifier from sklearn ensemble class.
• plt.plot(x, y): draw line plot for the values of y against x values.

Add notes from the video (PRs are welcome)

• Notes from Peter Ernicke

Key concepts

• scikit-learn
• Decision Trees

Related notes

• mlz-m01-model-selection-process
• mlz-m01-setting-up-the-environment
• mlz-m01-summary
• mlz-m02-explore-more
• mlz-m03-explore-more
• mlz-m03-feature-importance-mutual-information
• mlz-m03-one-hot-encoding
• mlz-m03-setting-up-the-validation-framework
• mlz-m03-summary
• mlz-m03-training-logistic-regression-with-scikit-learn
• mlz-m04-accuracy-and-dummy-model
• mlz-m04-cross-validation
• mlz-m04-roc-auc
• mlz-m04-roc-curves
• mlz-m05-python-virtual-environment-pipenv
• mlz-m05-summary
• mlz-m06-credit-risk-scoring-project
• mlz-m06-data-cleaning-and-preparation
• mlz-m06-decision-tree-learning-algorithm
• mlz-m06-decision-trees
• mlz-m06-decision-trees-parameter-tuning
• mlz-m06-explore-more
• mlz-m06-gradient-boosting-and-xgboost
• mlz-m06-selecting-the-best-model
• mlz-m06-summary
• mlz-m06-xgboost-parameter-tuning
• mlz-m09-explore-more

Sources

• Video
• Lesson file