When dealing with imbalanced classification tasks, where the number of instances in each class is significantly different, XGBoost offers several parameters to help prevent the model from overfitting on the majority class.
One such parameter is max_delta_step, which limits the maximum change in the predictions between each boosting step.
This example demonstrates how to manually tune the max_delta_step parameter and evaluate its impact on various classification metrics using a synthetic imbalanced dataset.
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
import numpy as np
# Generate an imbalanced synthetic dataset
X, y = make_classification(n_samples=1000, n_classes=2, weights=[0.9, 0.1], random_state=42)
# Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
def evaluate_model(max_delta_step):
model = XGBClassifier(n_estimators=100, max_delta_step=max_delta_step, random_state=42)
model.fit(X_train, y_train)
pred = model.predict(X_test)
return {
'max_delta_step': max_delta_step,
'accuracy': accuracy_score(y_test, pred),
'precision': precision_score(y_test, pred),
'recall': recall_score(y_test, pred),
'f1': f1_score(y_test, pred)
}
max_delta_step_values = [1, 3, 5, 7, 10]
results = []
for mds in max_delta_step_values:
results.append(evaluate_model(mds))
print(f"{'max_delta_step':<15}{'accuracy':<10}{'precision':<10}{'recall':<10}{'f1':<10}")
print('-'*55)
for res in results:
print(f"{res['max_delta_step']:<15}{res['accuracy']:<10.3f}{res['precision']:<10.3f}{res['recall']:<10.3f}{res['f1']:<10.3f}")
In this example, we generate a synthetic imbalanced dataset using make_classification from scikit-learn with a class distribution of 90% for class 0 and 10% for class 1.
We define a function evaluate_model that takes a max_delta_step value, initializes an XGBClassifier with the given value, fits the model on the training data, makes predictions on the test data, and returns a dictionary containing the max_delta_step value along with the computed accuracy, precision, recall, and F1-score.
We then define a list max_delta_step_values containing the values we want to test (1, 3, 5, 7, 10) and iterate over these values using a for loop. In each iteration, we call the evaluate_model function with the current max_delta_step value and append the results to the results list.
Finally, we print a table summarizing the performance metrics for each max_delta_step value.
The max_delta_step parameter controls the maximum allowed change in the predictions between each boosting step. By limiting this change, the model becomes more conservative and less likely to overfit on the majority class. However, setting max_delta_step too low may result in underfitting.
When tuning max_delta_step, it’s important to consider the balance between the model’s ability to learn from the minority class and its tendency to overfit on the majority class. The optimal value depends on the specific characteristics of your dataset and the trade-off between precision and recall that you are willing to make.
In this example, you can observe how different max_delta_step values affect the model’s performance. Generally, as max_delta_step increases, the model becomes more flexible and may achieve higher recall at the cost of lower precision. The F1-score provides a balanced measure of both precision and recall.
Based on the results, you can select an appropriate max_delta_step value that aligns with your project’s requirements. If precision is more important, you may prefer a lower value, while if recall is the priority, a higher value may be more suitable.