When working with imbalanced classification datasets or scenarios where accurately predicting the minority class is more important, the gamma deviance can be a useful evaluation metric for XGBoost models.
Gamma deviance measures the difference in log-likelihood between the model’s predictions and the ground truth labels. By placing more emphasis on correctly classifying the minority class, it provides a more informative assessment of model performance than traditional accuracy metrics.
To use gamma deviance as the evaluation metric in XGBoost, set the eval_metric parameter to 'gamma-deviance' when creating your model. This allows you to monitor the model’s performance on this metric during training and enables early stopping based on gamma deviance improvement.
Here’s an example of how to configure gamma deviance as the evaluation metric for an XGBoost classifier using scikit-learn:
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
import matplotlib.pyplot as plt
# Generate an imbalanced synthetic classification dataset
X, y = make_classification(n_samples=1000, n_classes=2, weights=[0.9, 0.1], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Create an XGBClassifier with gamma deviance as the evaluation metric
model = XGBClassifier(n_estimators=100, eval_metric='gamma-deviance', early_stopping_rounds=10, random_state=42)
# Train the model with early stopping
model.fit(X_train, y_train, eval_set=[(X_test, y_test)])
# Retrieve the gamma deviance values from the training process
results = model.evals_result()
epochs = len(results['validation_0']['gamma-deviance'])
x_axis = range(0, epochs)
# Plot the gamma deviance values
plt.figure()
plt.plot(x_axis, results['validation_0']['gamma-deviance'], label='Test')
plt.legend()
plt.xlabel('Number of Boosting Rounds')
plt.ylabel('Gamma Deviance')
plt.title('XGBoost Gamma Deviance Performance')
plt.show()
In this example, we generate an imbalanced synthetic classification dataset using scikit-learn’s make_classification function, with a 9:1 ratio between the majority and minority classes. We then split the data into training and testing sets.
We create an instance of XGBClassifier and set eval_metric='gamma-deviance' to specify gamma deviance as the evaluation metric. We also set early_stopping_rounds=10 to enable early stopping if the gamma deviance doesn’t improve for 10 consecutive rounds.
During training, we pass the testing set as the eval_set to monitor the model’s performance on unseen data. After training, we retrieve the gamma deviance values using the evals_result() method.
Finally, we plot the gamma deviance values against the number of boosting rounds to visualize the model’s performance during training. This plot helps us assess how well the model is learning to predict the minority class and determines the optimal number of boosting rounds.
By using gamma deviance as the evaluation metric, we can effectively monitor the model’s classification performance on imbalanced datasets, prevent overfitting through early stopping, and select the best model based on the lowest gamma deviance value.