When evaluating machine learning models, it’s often necessary to compare the performance of different algorithms to determine which one is best suited for a given task.
In this example, we’ll compare the performance of XGBoost and Random Forest classifiers on a binary classification problem and use a statistical significance test to assess whether the observed difference is meaningful or just due to chance.
By using cross-validation to obtain performance estimates for each model and applying a Student’s t-test to calculate a p-value, we can make an informed decision about which model to choose based on their performance and the statistical significance of the difference between them.
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
import xgboost as xgb
from scipy import stats
import numpy as np
# Generate a synthetic binary classification dataset
X, y = make_classification(n_samples=1000, n_classes=2, n_features=20, n_informative=10, random_state=42)
# Define an XGBoost classifier and a Random Forest classifier
xgb_clf = xgb.XGBClassifier(n_estimators=100, random_state=42)
rf_clf = RandomForestClassifier(n_estimators=100, random_state=42)
# Set the number of cross-validation folds and the significance level (alpha)
cv_folds = 5
alpha = 0.05
# Perform cross-validation for each model
xgb_scores = cross_val_score(xgb_clf, X, y, cv=cv_folds, scoring='accuracy')
rf_scores = cross_val_score(rf_clf, X, y, cv=cv_folds, scoring='accuracy')
# Calculate the mean and standard deviation of the accuracy scores for each model
xgb_mean, xgb_std = np.mean(xgb_scores), np.std(xgb_scores)
rf_mean, rf_std = np.mean(rf_scores), np.std(rf_scores)
# Apply a Student's t-test to the performance samples to calculate a p-value
_, p_value = stats.ttest_ind(xgb_scores, rf_scores)
# Compare the p-value to the configured significance level alpha to determine statistical significance
is_significant = p_value < alpha
# Print out the mean accuracy and standard deviation for each model, the p-value, and whether the difference is statistically significant
print(f"XGBoost mean accuracy: {xgb_mean:.4f} ± {xgb_std:.4f}")
print(f"Random Forest mean accuracy: {rf_mean:.4f} ± {rf_std:.4f}")
print(f"p-value: {p_value:.4f}")
print(f"The difference is {'statistically significant' if is_significant else 'not statistically significant'} at alpha = {alpha}")
In this example, we generate a synthetic binary classification dataset using make_classification from scikit-learn. We then define an XGBoost classifier (xgb_clf) and a Random Forest classifier (rf_clf) with default hyperparameters.
We set the number of cross-validation folds (cv_folds) and the significance level (alpha) for the statistical test. We perform k-fold cross-validation for each model using cross_val_score from scikit-learn, which returns an array of accuracy scores for each fold.
Next, we calculate the mean and standard deviation of the accuracy scores for each model. To determine if the difference in performance is statistically significant, we apply a Student’s t-test using ttest_ind from scipy.stats, which returns the t-statistic and the p-value.
Finally, we compare the p-value to the configured significance level alpha to determine if the difference is statistically significant. We print out the mean accuracy and standard deviation for each model, the p-value, and whether the difference is statistically significant based on the chosen alpha.
By following this approach, we can assess the performance difference between XGBoost and Random Forest classifiers and make an informed decision about which model to choose based on their performance and the statistical significance of the difference between them.