Course Wiki

Decision trees parameter tuning — Machine Learning Zoomcamp Module 6

In this lesson, we will discuss about different parameters used to control a Decision Tree (DT). Two of them, max_depth and min_samples_leaf have a greater importance than the others. We will further see how we first tune max_depth...

Machine Learning Zoomcamp › Module 6: Decision Trees and Ensemble Learning › Decision trees parameter tuning

Notes

In this lesson, we will discuss about different parameters used to control a Decision Tree (DT). Two of them, max_depth and min_samples_leaf have a greater importance than the others. We will further see how we first tune max_depth parameter and then move to tuning other parameters will help. After that, a dataframe will be created with all possible combinations of max_depth, min_sample_leaf and the auc score corresponding to them. These results will be visualized using a heatmap by pivoting the dataframe to easily determine the best possible max_depth and min_samples_leaf combination. Finally, the DT will be retrained using the identified parameter combination. The DT so trained will be viewed as a tree diagram, for visualizing decision rules.

Steps

• Fine-Tuning Process: iterate to find optimal parameter settings.

  • Start by tuning max_depth with various values to determine a subset of optimal depths.
  • Then, using this subset, fine-tune the model further by exploring different min_samples_leaf values.

  This method is computationally efficient for large datasets, though it may not be optimal for smaller ones.

• Heatmaps for Visualization: Store the scores (e.g., AUC) obtained during tuning in a pivot table, and create a heatmap with seaborn to easily identify high score areas, which helps pinpoint the optimal max_depth and min_samples_leaf combination.

NB: Choose parameter values that effectively control the tree’s size and avoid values like ‘nan’ (Not a Number), even if they seem to lead to better scores.

Importance of max_depth and min_samples_leaf

• Controlling Overfitting: these parameters play a critical role in preventing overfitting.

  • max_depth limits the tree’s complexity, preventing it from growing too deep and memorizing the training data.
  • min_samples_leaf ensures that leaf nodes have a sufficient number of samples, reducing the chance of creating nodes that are too specific to the training data.

• Impact on Bias and Variance: They also affect the model’s bias and variance.

  • Increasing max_depth and decreasing min_samples_leaf can lead to a more complex model with lower bias but higher variance.
  • Decreasing max_depth and increasing min_samples_leaf results in a simpler model with higher bias but lower variance.

It’s then important to find the right balance between max_depth and min_samples_leaf to achieve optimal model performance. This involves a trade-off between bias and variance, and the best values depend on the specific dataset and problem.

Add notes from the video (PRs are welcome)

• Notes from Peter Ernicke

Key concepts

• Pandas
• Classification Metrics
• Decision Trees

Related notes

• mlz-m01-introduction-to-pandas
• mlz-m01-model-selection-process
• mlz-m01-setting-up-the-environment
• mlz-m01-summary
• mlz-m02-car-price-prediction-project
• mlz-m02-categorical-variables
• mlz-m02-computing-rmse-on-validation-data
• mlz-m02-data-preparation
• mlz-m02-exploratory-data-analysis
• mlz-m02-feature-engineering
• mlz-m02-root-mean-squared-error-rmse
• mlz-m02-setting-up-the-validation-framework
• mlz-m02-using-the-model
• mlz-m03-data-preparation
• mlz-m03-eda
• mlz-m03-feature-importance-churn-rate-and-risk-ratio
• mlz-m03-feature-importance-correlation
• mlz-m03-feature-importance-mutual-information
• mlz-m03-setting-up-the-validation-framework
• mlz-m04-evaluation-metrics-session-overview
• mlz-m04-explore-more
• mlz-m04-precision-and-recall
• mlz-m04-roc-auc
• mlz-m04-roc-curves
• mlz-m04-summary
• mlz-m05-summary
• mlz-m06-credit-risk-scoring-project
• mlz-m06-data-cleaning-and-preparation
• mlz-m06-decision-tree-learning-algorithm
• mlz-m06-decision-trees
• mlz-m06-ensemble-learning-and-random-forest
• mlz-m06-explore-more
• mlz-m06-gradient-boosting-and-xgboost
• mlz-m06-selecting-the-best-model
• mlz-m06-summary
• mlz-m06-xgboost-parameter-tuning

Sources

• Video
• Lesson file