Early stopping is a valuable technique in XGBoost that prevents overfitting by halting the training process when the model’s performance on a validation set stops improving.
By default, XGBoost models in the scikit-learn API will automatically use the iteration_range to specify the best_iteration so that all predictions are made using the best model found using early stopping.
Nevertheless, we may want to manually specify the iteration_range for the best model found with early stopping.
Here is an example:
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
# Create a synthetic dataset for binary classification
X, y = make_classification(n_samples=1000, n_classes=2, n_features=20, random_state=42)
# Split the dataset into train, validation, and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=42)
# Define the XGBoost classifier with early stopping
xgb_model = XGBClassifier(n_estimators=1000, early_stopping_rounds=10, eval_metric='error')
# Train the model with early stopping and store the best model
xgb_model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=True)
# Use the best model to make predictions on the test set
y_pred_auto = xgb_model.predict(X_test)
# Print the classification accuracy of the best model on the test set
accuracy = (y_pred_auto == y_test).mean()
print(f"Automatic Best Model Test Accuracy: {accuracy:.4f}")
# Manually use the best model to make predictions on the test set
y_pred_manual = xgb_model.predict(X_test, iteration_range=(0, xgb_model.best_iteration + 1))
# Print the classification accuracy of the best model on the test set
accuracy = (y_pred_manual == y_test).mean()
print(f"Manual Best Model Test Accuracy: {accuracy:.4f}")
In this example, we first create a synthetic binary classification dataset using make_classification() from scikit-learn. We then split the data into train, validation, and test sets.
Next, we define an XGBClassifier with early stopping enabled by setting early_stopping_rounds=10. We also specify eval_metric='error' to use classification error as the evaluation metric.
During training, the best model is found via early stopping and the boosting iteration (round) is stored in the model’s best_iteration property.
After training, we use the scikit-learn API’s ability to automatically use the best model found during early sotpping to make a prediction on the test set and report the accuracy.
Finally, we manually specify the iteration_range for the model using the best_iteration and make a prediction on the test set and report accuracy.
The accuracy from the two ways of using the best model to make predictions match.
By using early stopping and retrieving the best model, we ensure that we get the most optimal model for our dataset, avoiding overfitting and potentially improving performance on unseen data.