The normalize_type parameter is an important setting when using the XGBoost Dart booster, which can be specified by setting booster='dart'.
This parameter determines the type of normalization algorithm used for the weights of dropped trees and newly added trees during the boosting process.
XGBoost Dart supports two normalization algorithms:
'tree': New trees have the same weight as each of the dropped trees. The weight of new trees is 1 / (k + learning_rate), where k is the number of dropped trees. Dropped trees are scaled by a factor of k / (k + learning_rate).'forest': New trees have the same weight as the sum of the dropped trees (forest). The weight of new trees is 1 / (1 + learning_rate). Dropped trees are scaled by a factor of 1 / (1 + learning_rate).
Choosing the appropriate normalization type can impact the model’s performance and convergence. Here’s an example demonstrating how to set the normalize_type parameter and compare the performance of models using different normalization algorithms:
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score
# Generate a synthetic multiclass classification dataset
X, y = make_classification(n_samples=1000, n_features=10, n_classes=3, n_informative=8, random_state=42)
# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Initialize an XGBClassifier with Dart booster and normalize_type='tree'
clf_tree_norm = XGBClassifier(booster='dart', max_depth=5, learning_rate=0.1, n_estimators=100,
rate_drop=0.2, normalize_type='tree', random_state=42)
# Initialize an XGBClassifier with Dart booster and normalize_type='forest'
clf_forest_norm = XGBClassifier(booster='dart', max_depth=5, learning_rate=0.1, n_estimators=100,
rate_drop=0.2, normalize_type='forest', random_state=42)
# Train the models
clf_tree_norm.fit(X_train, y_train)
clf_forest_norm.fit(X_train, y_train)
# Make predictions on the test set
pred_tree_norm = clf_tree_norm.predict(X_test)
pred_forest_norm = clf_forest_norm.predict(X_test)
# Evaluate the models
accuracy_tree_norm = accuracy_score(y_test, pred_tree_norm)
accuracy_forest_norm = accuracy_score(y_test, pred_forest_norm)
print(f"Accuracy (normalize_type='tree'): {accuracy_tree_norm:.4f}")
print(f"Accuracy (normalize_type='forest'): {accuracy_forest_norm:.4f}")
In this example, we generate a synthetic multiclass classification dataset and split it into training and testing sets. We then initialize two XGBClassifier instances with the Dart booster, one with normalize_type='tree' and another with normalize_type='forest'. All other parameters, such as max_depth, learning_rate, and rate_drop, are kept the same between the two models.
After training both models, we make predictions on the test set and evaluate their accuracies. The output will show the difference in performance between the model using the “tree” normalization algorithm and the model using the “forest” normalization algorithm.
By comparing the accuracies, you can observe the impact of the normalize_type parameter on the model’s performance. The optimal choice of normalization type may depend on the specific dataset and problem at hand.
When using the XGBoost Dart booster, it’s recommended to experiment with both normalize_type options along with other hyperparameters to find the best configuration for your specific use case. Keep in mind that the choice of normalization algorithm can affect the model’s convergence and the interpretability of the feature importances.