Course Wiki

XGBoost parameter tuning — Machine Learning Zoomcamp Module 6

XGBoost has various tunable parameters but the three most important ones are:

mlz-module-06

Machine Learning Zoomcamp › Module 6: Decision Trees and Ensemble Learning › XGBoost parameter tuning

Notes

XGBoost has various tunable parameters but the three most important ones are:

eta (default=0.3)
- It is also called learning_rate and is used to prevent overfitting by regularizing the weights of new features in each boosting step. range: [0, 1]
max_depth (default=6)
- Maximum depth of a tree. Increasing this value will make the model more complex and more likely to overfit. range: [0, inf]
min_child_weight (default=1)
- Minimum number of samples in leaf node. range: [0, inf]

For XGBoost models, there are other ways of finding the best parameters as well but the one we implement in the notebook follows the sequence of:

First find the best value for eta
Second, find the best value for max_depth
Third, find the best value for min_child_weight

Other useful parameter are:

subsample (default=1)
- Subsample ratio of the training instances. Setting it to 0.5 means that model would randomly sample half of the training data prior to growing trees. range: (0, 1]
colsample_bytree (default=1)
- This is similar to random forest, where each tree is made with the subset of randomly choosen features.
lambda (default=1)
- Also called reg_lambda. L2 regularization term on weights. Increasing this value will make model more conservative.
alpha (default=0)
- Also called reg_alpha. L1 regularization term on weights. Increasing this value will make model more conservative.

Alternative: Tuning XGBoost using a loop

Instead of repeating the training process manually for each eta value, you can use a loop to automate it (and other parameters to be tuned):

# Train XGBoost models for each eta and store AUC results
scores = {} # dictionary to store results for each eta
etas = [0.01, 0.05, 0.1, 0.3, 1.0] # list of parameter values. in this case it is 'eta'.

for eta in etas:
    evals_result = {}

    xgb_params = {
        'eta': eta,
        'max_depth': 6,
        'min_child_weight': 1,

        'objective': 'binary:logistic',
        'eval_metric': 'auc',

        'nthread': 8,
        'seed': 1,
        'verbosity': 1
    }

    model = xgb.train(
        xgb_params,
        dtrain,
        evals=watchlist,
        verbose_eval=0,
        num_boost_round=200,
        evals_result=evals_result
    )

    columns = ['iter', 'train_auc', 'val_auc']
    train_aucs = list(evals_result['train'].values())[0]
    val_aucs = list(evals_result['val'].values())[0]

    df_results = pd.DataFrame(
        list(zip(range(1, len(train_aucs) + 1), train_aucs, val_aucs)),
        columns=columns
    )

    key = f'eta={eta}'
    scores[key] = df_results

Add notes from the video (PRs are welcome)

⚠️	The notes are written by the community. If you see an error here, please create a PR with a fix.

DataTalks.Club

XGBoost parameter tuning — Machine Learning Zoomcamp Module 6

Related Pages

Notes

Alternative: Tuning XGBoost using a loop

Key concepts

Sources

DataTalks.Club

XGBoost parameter tuning — Machine Learning Zoomcamp Module 6

Related Pages

Notes

Alternative: Tuning XGBoost using a loop

Key concepts

Related notes

Sources