XGBoost’s feature importances are a powerful tool for understanding the relative contribution of each feature to the model’s predictions.
It’s crucial to be aware that these importances can be stable, even if features are removed.
In this example, we’ll demonstrate how removing features does not dramatically alter the feature importances of an XGBoost model. We’ll use a synthetic dataset to illustrate this effect and highlight the importance of considering the stability of feature importances when interpreting XGBoost models.
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import xgboost as xgb
import numpy as np
import matplotlib.pyplot as plt
# Generate a synthetic dataset with 10 features, 5 of which are informative
X, y = make_classification(n_samples=1000, n_features=10, n_informative=5,
n_redundant=0, n_repeated=0, random_state=42)
# Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Train an XGBoost model on the full dataset
model_full = xgb.XGBClassifier(n_estimators=100, learning_rate=0.1, random_state=42)
model_full.fit(X_train, y_train)
# Get feature importances for the full dataset
importances_full = model_full.feature_importances_
# Remove the first few features from the dataset
X_train_reduced = X_train[:, 3:]
X_test_reduced = X_test[:, 3:]
# Train an XGBoost model on the reduced dataset
model_reduced = xgb.XGBClassifier(n_estimators=100, learning_rate=0.1, random_state=42)
model_reduced.fit(X_train_reduced, y_train)
# Get feature importances for the reduced dataset
importances_reduced = model_reduced.feature_importances_
# Visualize the feature importances before and after feature removal
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 6))
ax1.bar(range(len(importances_full)), importances_full)
ax1.set_title("Feature Importances (Full Dataset)")
ax1.set_xlabel("Feature Index")
ax1.set_ylabel("Importance")
ax2.bar(range(len(importances_reduced)), importances_reduced)
ax2.set_title("Feature Importances (Reduced Dataset)")
ax2.set_xlabel("Feature Index")
ax2.set_ylabel("Importance")
plt.tight_layout()
plt.show()
In this example, we generate a synthetic dataset with 10 features, 5 of which are informative. We train an XGBoost classifier on the full dataset and compute the feature importances. We then remove the first few features from the dataset and retrain the model on the reduced dataset, computing the feature importances for the reduced model.
Finally, we visualize the feature importances for both the full and reduced datasets using bar plots.
The resulting plots will show no dramatic changes in feature importances when features are removed from the dataset.