This example will differentiate between the XGBoost objectives “multi:softmax” and “multi:softprob,” which are both used for multi-class classification tasks.
It will explain when to use each objective, providing a full code example to highlight their implementation and key differences.
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score, log_loss
# Generate a synthetic multi-class classification dataset
X, y = make_classification(n_samples=1000, n_clusters_per_class=1, n_classes=3, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Define the model using the multi:softmax objective
model_softmax = XGBClassifier(objective="multi:softmax", num_class=3, n_estimators=100, learning_rate=0.1)
model_softmax.fit(X_train, y_train)
y_pred_softmax = model_softmax.predict(X_test)
accuracy_softmax = accuracy_score(y_test, y_pred_softmax)
# Define the model using the multi:softprob objective
model_softprob = XGBClassifier(objective="multi:softprob", num_class=3, n_estimators=100, learning_rate=0.1)
model_softprob.fit(X_train, y_train)
y_pred_softprob = model_softprob.predict_proba(X_test)
logloss_softprob = log_loss(y_test, y_pred_softprob)
print(f"Accuracy for 'multi:softmax': {accuracy_softmax:.4f}")
print(f"Log Loss for 'multi:softprob': {logloss_softprob:.4f}")
Analysis of Outcomes:
- The “multi:softmax” objective returns discrete class labels for the predictions and is best when specific class assignments are needed.
- The “multi:softprob” provides a probability for each class, which is useful when you need to understand the confidence level of the predictions.
Best Practices and Tips:
- Choosing the right objective is crucial depending on the nature of your classification task. Use “multi:softmax” for a straightforward class prediction where only the class label is needed. Opt for “multi:softprob” when the probabilities of class assignments can influence decision-making or when needing to evaluate the confidence level of predictions.
- When tuning hyperparameters, adjusting
learning_rate,max_depth, andn_estimatorscan significantly affect model performance. Experiment with different settings to find the optimal configuration for your data. - Consider preprocessing your data, such as scaling or principal component analysis (PCA), to improve model performance. This is especially relevant when dealing with high-dimensional data or features on different scales.