XGBoosting Home | About | Contact | Examples

Out-of-Bag (OOB) Estimates of Performance for XGBoost

Out-of-bag (OOB) estimates provide an efficient and unbiased way to assess the performance of a bagged model without the need for a separate validation set.

Out-of-bag (OOB) estimates provide a convenient way to assess the performance of a bagged model without the need for a separate validation set.

Below provides an example of reporting an OOB model error using an evaluation set.

from sklearn.datasets import make_classification
from xgboost import XGBClassifier

# Generate a synthetic binary classification dataset
X, y = make_classification(n_samples=1000, n_classes=2, random_state=42)

# Train an XGBoost classifier with OOB estimates
xgb_clf = XGBClassifier(n_estimators=100, subsample=0.8, eval_metric='logloss')
xgb_clf.fit(X, y, eval_set=[(X, y)], verbose=False)

# Access the OOB metrics
oob_accuracy = 1 - xgb_clf.evals_result()['validation_0']['logloss'][-1]
print(f"OOB Accuracy: {oob_accuracy:.4f}")

Here’s how it works:

  1. We start by generating a synthetic binary classification dataset using scikit-learn’s make_classification function. In practice, you would use your actual training data.

  2. We create an instance of the XGBClassifier with the following parameters:

    • n_estimators=100: The number of boosting rounds.
    • subsample=0.8: The fraction of samples to be used for fitting each tree. This enables the OOB estimates.
    • eval_metric='logloss': The evaluation metric to be used. In this case, we use log loss, which is suitable for binary classification.

  3. We fit the XGBoost classifier using the fit method, specifying the training data (X and y) and the evaluation set (eval_set). Setting verbose=False suppresses the training output.

  4. After training, we access the OOB metrics using the evals_result() method. In this example, we calculate the OOB accuracy by subtracting the final log loss value from 1.

OOB estimates work by leveraging the fact that each tree in the XGBoost model is trained on a random subset of the training data (controlled by the subsample parameter). The samples that are not used for training a particular tree are considered “out-of-bag” for that tree. These OOB samples are used to evaluate the performance of the tree, and the OOB predictions are aggregated across all trees to provide an overall estimate of the model’s performance.

By using OOB estimates, you can assess your XGBoost model’s performance without sacrificing any of your training data for validation. This is particularly useful when working with limited data or when you want to make the most of your available data for training.

It’s important to note that OOB estimates are only available when the subsample parameter is set to a value less than 1. If subsample=1 (default), all samples are used for training each tree, and there are no OOB samples available for evaluation.



See Also