The learning_rate parameter in XGBoost is an alias for the eta parameter, which controls the step size at each boosting iteration.
By adjusting the learning rate, you can influence the model’s performance and training time.
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
# Generate synthetic data
X, y = make_classification(n_samples=1000, n_features=20, n_informative=2, n_redundant=10, random_state=42)
# Split the dataset into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Initialize the XGBoost classifier with a lower learning rate (eta)
model = XGBClassifier(eta=0.01, eval_metric='logloss')
# Fit the model
model.fit(X_train, y_train)
# Make predictions
predictions = model.predict(X_test)
As discussed in the tip on configuring the eta parameter, the learning rate determines the contribution of each tree to the final outcome by scaling the weights of the features.
A lower learning rate can lead to better generalization and reduced overfitting, while a higher learning rate may result in faster learning but suboptimal solutions.
To recap, the key points when configuring the learning_rate parameter are:
- Valid range: 0 to 1, inclusive
- Default value: 0.3
- Impact on model performance:
- Lower values slow down learning but can improve generalization
- Higher values speed up learning but may lead to suboptimal results
- Interaction with the number of boosting rounds:
- Lower learning rates typically require more boosting rounds
- Higher learning rates may converge faster but require careful tuning of other parameters
For practical guidance on choosing the right learning rate value, refer to the tip on configuring the eta parameter.