XGBoost Configure fit() "verbose" Parameter

The verbose parameter in XGBoost’s fit() method controls the verbosity of the output during training.

Setting it to True will print the evaluation metric at each boosting stage when using early stopping, which can be helpful for monitoring the training process and identifying potential issues.

It is True by default, but it can be a good idea to set it explictly

Here’s how you can use the verbose parameter:

from xgboost import XGBRegressor
from sklearn.datasets import make_regression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

# Generate a synthetic dataset for regression
X, y = make_regression(n_samples=1000, n_features=10, noise=0.1, random_state=42)

# Split the data into train, validation, and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=42)

# Initialize an XGBRegressor with early stopping
model = XGBRegressor(n_estimators=1000, learning_rate=0.1, random_state=42, early_stopping_rounds=10)

# Fit the model with verbose=True to print the evaluation metric at each boosting stage
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=True)

# Make predictions on the same data
y_pred = model.predict(X_test)

# Print the mean squared error
mse = mean_squared_error(y_test, y_pred)
print(f"Mean Squared Error: {mse:.4f}")

In this example:

1. We generate a synthetic dataset for a regression problem using sklearn’s make_regression function.

2. We initialize an XGBRegressor with default hyperparameters and enable early stopping.

3. We fit the model on the training dataset with an eval_set for the validation data and verbose=True, which will print the evaluation metric (in this case, the default metric is RMSE) at each boosting stage.

4. We make predictions on the same data (just for demonstration purposes) and print the mean squared error to show the model’s performance.

By setting verbose=True, we can see the progress of the training process and the improvement of the evaluation metric at each boosting stage.

This can be useful for monitoring the training process and identifying if the model is converging or if it might be overfitting or underfitting. You can adjust the verbosity level by setting verbose to different positive integers, with higher values producing more detailed output.

This verbose output can be disabled by setting verbose=False.