When dealing with binary classification problems where the classes are imbalanced, the Area Under the Precision-Recall Curve (AUCPR) is a more informative evaluation metric than accuracy or AUC-ROC.
By setting eval_metric='aucpr' in XGBoost, you can monitor your model’s performance on the positive (minority) class during training, which is particularly useful when correctly identifying positive instances is more important than overall accuracy.
Here’s an example of how to use AUCPR as the evaluation metric with XGBoost and scikit-learn:
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
import matplotlib.pyplot as plt
# Generate a synthetic imbalanced binary classification dataset
X, y = make_classification(n_samples=1000, n_classes=2, weights=[0.9, 0.1], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Create an XGBClassifier with AUCPR as the evaluation metric
model = XGBClassifier(n_estimators=100, eval_metric='aucpr', scale_pos_weight=9, early_stopping_rounds=10, random_state=42)
# Train the model with early stopping
model.fit(X_train, y_train, eval_set=[(X_test, y_test)])
# Retrieve the AUCPR values from the training process
results = model.evals_result()
epochs = len(results['validation_0']['aucpr'])
x_axis = range(0, epochs)
# Plot the AUCPR values
plt.figure()
plt.plot(x_axis, results['validation_0']['aucpr'], label='Test')
plt.legend()
plt.xlabel('Number of Boosting Rounds')
plt.ylabel('AUCPR')
plt.title('XGBoost AUCPR Performance')
plt.show()
In this example, we generate a synthetic imbalanced binary classification dataset using scikit-learn’s make_classification function with a 9:1 class ratio. We then split the data into training and testing sets.
We create an instance of XGBClassifier and set eval_metric='aucpr' to specify AUCPR as the evaluation metric. We also set scale_pos_weight=9 to handle the class imbalance by assigning a higher weight to the minority class. The early_stopping_rounds=10 parameter enables early stopping if the AUCPR doesn’t improve for 10 consecutive rounds.
During training, we pass the testing set as the eval_set to monitor the model’s performance on unseen data. After training, we retrieve the AUCPR values using the evals_result() method.
Finally, we plot the AUCPR values against the number of boosting rounds to visualize the model’s performance during training. This plot helps us assess whether the model is overfitting or underfitting and determines the optimal number of boosting rounds.
By using AUCPR as the evaluation metric and handling class imbalance with scale_pos_weight, we can effectively train an XGBoost model that focuses on the performance of the positive (minority) class, which is crucial in many real-world applications such as fraud detection or disease diagnosis.