XGBoost Evaluate Model using the Jackknife Method (LOOCV)

The Jackknife method, also known as leave-one-out cross-validation, is a technique for assessing the performance of a machine learning model by iteratively training the model on all but one observation and then making a prediction for the held-out observation.

This process is repeated for each observation in the dataset, providing a robust estimate of the model’s performance.

In this example, we’ll demonstrate how to implement the Jackknife method to evaluate an XGBooster regressor using a synthetic dataset.

import numpy as np
from sklearn.datasets import make_regression
from sklearn.metrics import mean_squared_error
from xgboost import XGBRegressor

# Generate a synthetic dataset
X, y = make_regression(n_samples=100, n_features=5, noise=0.1, random_state=42)

# Create an XGBRegressor
model = XGBRegressor(n_estimators=100, learning_rate=0.1, random_state=42)

# Implement the Jackknife method
y_pred = []
for i in range(len(X)):
    X_train = np.concatenate((X[:i], X[i+1:]))
    y_train = np.concatenate((y[:i], y[i+1:]))
    model.fit(X_train, y_train)
    y_pred.append(model.predict(X[i].reshape(1, -1))[0])

# Calculate the Jackknife estimate of RMSE
jackknife_rmse = np.sqrt(mean_squared_error(y, y_pred))

print(f"Jackknife estimate of RMSE: {jackknife_rmse:.2f}")

Here’s how the code works:

1. We generate a synthetic dataset using scikit-learn’s make_regression() function, specifying the number of samples, features, and noise level.
2. We create an XGBRegressor with specified hyperparameters.
3. We implement the Jackknife method by iterating over the dataset, removing one observation at a time:
   • We train the model on the remaining data using model.fit().
   • We make a prediction for the held-out observation using model.predict().
   • We store the prediction in the y_pred list.
4. We calculate the Jackknife estimate of the model’s performance metric (in this case, RMSE) using the predicted values and the actual target values.
5. We print the Jackknife estimate of RMSE.

The Jackknife method provides a more conservative estimate of a model’s performance compared to a single train-test split, as it assesses the model’s performance on every possible subset of the data where one observation is left out. This can help identify if the model’s performance is overly dependent on specific observations and provide a more robust estimate of how the model will perform on unseen data.