While XGBoost is best known for its tree-based models, it also offers a unique feature that allows it to fit linear models. This can be useful for certain types of problems where a linear relationship is expected, or when model interpretability is important.
To configure XGBoost to use a linear model, set the booster parameter to 'gblinear'. The updater parameter is then used to specify the linear model algorithm. The available options are 'shotgun' for parallel coordinate descent, and 'coord_descent' for coordinate descent.
Here’s an example demonstrating how to configure an XGBoost linear model with the 'coord_descent' updater for a binary classification task using a synthetic dataset:
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score
# Generate a synthetic binary classification dataset
X, y = make_classification(n_samples=1000, n_features=10, random_state=42)
# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Initialize an XGBClassifier with linear booster and coord_descent updater
model = XGBClassifier(booster='gblinear', updater='coord_descent')
# Train the model
model.fit(X_train, y_train)
# Make predictions on the test set
predictions = model.predict(X_test)
# Evaluate the model
accuracy = accuracy_score(y_test, predictions)
print(f"Accuracy: {accuracy:.4f}")
In this example, we first generate a synthetic binary classification dataset using make_classification() from scikit-learn. We then split the data into training and testing sets.
Next, we initialize an XGBClassifier with booster='gblinear' to specify a linear model, and updater='coord_descent' to use the coordinate descent algorithm.
We then train the model using the fit() method, make predictions on the test set using predict(), and evaluate the model’s performance using accuracy_score().
Using a linear model can be beneficial when you expect a linear relationship between the features and the target, or when you need a more interpretable model. However, for many complex real-world problems, tree-based models often outperform linear models.
As with any model, it’s important to evaluate the performance and compare it with other approaches to determine if a linear model is suitable for your specific problem. If you’re unsure which updater to use, 'coord_descent' is generally a good default choice.