Course Wiki

Decision trees — Machine Learning Zoomcamp Module 6

Decision Trees are powerful algorithms, capable of fitting complex datasets. The decision trees make predictions based on the bunch of if/else statements by splitting a node into two or more sub-nodes.

Machine Learning Zoomcamp › Module 6: Decision Trees and Ensemble Learning › Decision trees

Notes

Decision Trees are powerful algorithms, capable of fitting complex datasets. The decision trees make predictions based on the bunch of if/else statements by splitting a node into two or more sub-nodes.

With versatility, the decision tree is also prone to overfitting. One of the reasons why this algorithm often overfits is its depth. It tends to memorize all the patterns in the train data but struggles to perform well on the unseen data (validation or test set).

To overcome the overfitting problem, we can reduce the complexity of the algorithm by reducing the depth size.

A decision tree with a depth of 1 is called decision stump and has only one split from the root.

Classes, functions, and methods:

• DecisionTreeClassifier: classification model from sklearn.tree class.
• max_depth: hyperparameter to control the maximum depth of decision tree algorithm.
• export_text: method from sklearn.tree class to display the text report showing the rules of a decision tree.

Note: we have already covered DictVectorizer in session 3 and roc_auc_score in session 4 respectively.

Add notes from the video (PRs are welcome)

• Notes from Peter Ernicke
• Notes from Peter Ernicke

Key concepts

• scikit-learn
• Decision Trees

Related notes

• mlz-m01-model-selection-process
• mlz-m01-setting-up-the-environment
• mlz-m01-summary
• mlz-m02-explore-more
• mlz-m03-explore-more
• mlz-m03-feature-importance-mutual-information
• mlz-m03-one-hot-encoding
• mlz-m03-setting-up-the-validation-framework
• mlz-m03-summary
• mlz-m03-training-logistic-regression-with-scikit-learn
• mlz-m04-accuracy-and-dummy-model
• mlz-m04-cross-validation
• mlz-m04-roc-auc
• mlz-m04-roc-curves
• mlz-m05-python-virtual-environment-pipenv
• mlz-m05-summary
• mlz-m06-credit-risk-scoring-project
• mlz-m06-data-cleaning-and-preparation
• mlz-m06-decision-tree-learning-algorithm
• mlz-m06-decision-trees-parameter-tuning
• mlz-m06-ensemble-learning-and-random-forest
• mlz-m06-explore-more
• mlz-m06-gradient-boosting-and-xgboost
• mlz-m06-selecting-the-best-model
• mlz-m06-summary
• mlz-m06-xgboost-parameter-tuning
• mlz-m09-explore-more

Sources

• Video
• Lesson file