The eval_set parameter in XGBoost allows users to specify validation sets for monitoring model performance during training.
This parameter is used in the scikit-learn fit() function. The equivalent is the evals argument in the XGBoost train() function in the native API.
It is commonly used in conjunction with early stopping to prevent overfitting and optimize the number of boosting rounds.
The eval_metric model parameter defines the performance metric to calculate on the eval_set each boosting round. Results of the eval_metric on the eval_set can be retrieved via the evals_result() method on the model.
This example will demonstrate how to configure and use the eval_set parameter with and without early stopping.
Use Evaluation Set
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from xgboost import XGBRegressor
# Load the California Housing dataset
housing = fetch_california_housing()
X, y = housing.data, housing.target
# Split the data into training, validation, and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=42)
# Configure an XGBRegressor model with the "eval_set" parameter
model = XGBRegressor(n_estimators=1000, learning_rate=0.1, subsample=0.8, colsample_bytree=0.8, random_state=42)
# Train the model using the "fit()" method with the training data and "eval_set"
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
# Make predictions using the trained model on the test set
y_pred = model.predict(X_test)
# Print the model's performance metrics (e.g., RMSE, MAE) on the test set
rmse = np.sqrt(np.mean((y_test - y_pred)**2))
mae = np.mean(np.abs(y_test - y_pred))
print(f"Test RMSE: {rmse:.4f}")
print(f"Test MAE: {mae:.4f}")
Use Evaluation Set For Early Stopping
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from xgboost import XGBRegressor
# Load the California Housing dataset
housing = fetch_california_housing()
X, y = housing.data, housing.target
# Split the data into training, validation, and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=42)
# Configure another XGBRegressor model with early stopping
model_es = XGBRegressor(n_estimators=1000, learning_rate=0.1, subsample=0.8, colsample_bytree=0.8, early_stopping_rounds=10, random_state=42)
# Train the model with early stopping
model_es.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
# Access the best iteration when using early stopping
best_iteration = model_es.best_iteration
print(f"Best iteration: {best_iteration}")
# Make predictions using the best model
y_pred_es = model_es.predict(X_test, iteration_range=(0, best_iteration))
# Print the model's performance metrics (e.g., RMSE, MAE) on the test set
rmse_es = np.sqrt(np.mean((y_test - y_pred_es)**2))
mae_es = np.mean(np.abs(y_test - y_pred_es))
print(f"Test RMSE (with early stopping): {rmse_es:.4f}")
print(f"Test MAE (with early stopping): {mae_es:.4f}")
By configuring the eval_set parameter with a validation set, XGBoost can monitor the model’s performance on unseen data during training. This enables users to track the model’s progress and identify potential overfitting.
When combined with early stopping, the eval_set parameter helps determine the optimal number of boosting rounds. Early stopping halts the training process when the model’s performance on the validation set stops improving for a specified number of consecutive iterations (early_stopping_rounds).
To access the best model when using early stopping, use the best_iteration attribute, which stores the iteration at which the model achieved the best performance on the validation set. When making predictions, specify the iteration_range parameter in the predict() method to ensure that predictions are made using the best model.
By leveraging the eval_set parameter and early stopping, users can optimize their XGBoost models, prevent overfitting, and improve generalization performance.