Predict Calibrated Probabilities with XGBoost

While XGBoost is a powerful and widely used algorithm for classification tasks, the predicted probabilities it outputs may not always be well-calibrated out-of-the-box.

Calibration refers to the process of adjusting the predicted probabilities to better align with the true likelihood of an event occurring. By calibrating your XGBoost model, you can improve the reliability and interpretability of its predictions, which is particularly important in applications where the actual probability values matter, such as risk assessment or cost-sensitive decision making.

Fortunately, scikit-learn provides a convenient way to calibrate the probabilities of any classifier, including XGBoost, through the CalibratedClassifierCV class. Here’s how you can use it:

from xgboost import XGBClassifier
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Generate synthetic binary classification dataset
X, y = make_classification(n_samples=1000, n_classes=2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train XGBoost classifier
xgb_clf = XGBClassifier(random_state=42)
xgb_clf.fit(X_train, y_train)

# Calibrate probabilities using CalibratedClassifierCV
calibrated_clf = CalibratedClassifierCV(estimator=xgb_clf, method='sigmoid', cv=5)
calibrated_clf.fit(X_train, y_train)

# Compare uncalibrated and calibrated probabilities
uncalibrated_probs = xgb_clf.predict_proba(X_test)
calibrated_probs = calibrated_clf.predict_proba(X_test)

print("Uncalibrated probabilities:", uncalibrated_probs[:5])
print("Calibrated probabilities:", calibrated_probs[:5])

Here’s a step-by-step breakdown:

1. First, we initialize an XGBoost classifier (XGBClassifier) and train it on our data. In this example, we’re using a synthetic binary classification dataset generated by scikit-learn’s make_classification function.

2. Next, we wrap our trained XGBoost model in the CalibratedClassifierCV class. We specify the base estimator (our XGBoost model), the calibration method ('sigmoid' or 'isotonic'), and the number of cross-validation folds to use for calibration (cv=5). The 'sigmoid' method is a good default choice, but you can also try 'isotonic' and see which works better for your data.

3. We then fit the calibrated classifier on our training data. During this process, CalibratedClassifierCV will use cross-validation to estimate the calibration map and adjust the predicted probabilities accordingly.

4. Finally, we can compare the uncalibrated probabilities (directly from the XGBoost model) with the calibrated probabilities. In this example, we simply print out the first five probability pairs for illustration.

By calibrating your XGBoost model, you can have more confidence in the predicted probabilities it outputs, leading to better-informed decisions and more accurate risk assessments. Keep in mind that calibration is especially useful when the probability values themselves are important, rather than just the final class predictions.