XGBoosting Home | About | Contact | Examples

Configure XGBoost "reg:logistic" Objective

The "reg:logistic" objective in XGBoost is used for regression or binary classification tasks, where the goal is to predict the probability of an instance belonging to the positive class.

This objective optimizes the log loss (also known as binary cross-entropy) between the predicted probabilities and the true labels, making it a suitable choice when the target variable has two classes, and the model should output class probabilities.

“reg:logistic” Objective For Binary Classification

Here is an example of using the "reg:logistic" objective to model class probabilities.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
from sklearn.metrics import log_loss, accuracy_score

# Generate a synthetic dataset for binary classification
X, y = make_classification(n_samples=1000, n_classes=2, n_features=10, random_state=42)

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize an XGBClassifier with the "reg:logistic" objective
model = XGBClassifier(objective="reg:logistic", n_estimators=100, learning_rate=0.1)

# Fit the model on the training data
model.fit(X_train, y_train)

# Make predictions on the test set
y_pred_proba = model.predict_proba(X_test)[:, 1]  # Predicted probabilities for the positive class
y_pred = model.predict(X_test)

# Calculate the log loss and accuracy of the predictions
log_loss_value = log_loss(y_test, y_pred_proba)
accuracy = accuracy_score(y_test, y_pred)

print(f"Log Loss: {log_loss_value:.4f}")
print(f"Accuracy: {accuracy:.4f}")

“reg:logistic” Objective For Regression

Here is an example of using the "reg:logistic" objective for regression to to model general probabilities.

from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from xgboost import XGBRegressor
import numpy as np

# Generate a synthetic dataset for regression
X, y = make_regression(n_samples=1000, n_features=10, noise=0.1, random_state=42)

# Normalize y to be between 0 and 1, simulating probability-like continuous outcomes
y = (y - y.min()) / (y.max() - y.min())

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize an XGBRegressor with the "reg:logistic" objective
model = XGBRegressor(objective="reg:logistic", n_estimators=100, learning_rate=0.1)

# Fit the model on the training data
model.fit(X_train, y_train)

# Make predictions on the test set
y_pred = model.predict(X_test)

# Calculate the Mean Squared Error of the predictions
mse = mean_squared_error(y_test, y_pred)

print(f"Mean Squared Error: {mse:.4f}")

When using the "reg:logistic" objective for regression or binary classification, consider the following tips:

  1. Ensure that the target variable is binary, with labels typically represented as 0 and 1.
  2. Scale the input features to a similar range to improve convergence and model performance.
  3. Use appropriate evaluation metrics for binary classification, such as log loss and accuracy, to assess the model’s performance.
  4. Tune hyperparameters like max_depth, learning_rate, and n_estimators to optimize performance.
  5. Access the predicted probabilities for the positive class using model.predict_proba(X)[:, 1].
  6. Consider setting a probability threshold for making class predictions based on domain-specific requirements or by optimizing metrics like F1-score or precision-recall curves.

By understanding and properly configuring the "reg:logistic" objective in XGBoost, data scientists and machine learning engineers can effectively tackle binary classification problems and obtain well-calibrated probability estimates.



See Also