Spot why feeding hard labels to ROC-AUC and PR-AUC silently breaks evaluation on a highly imbalanced classifier.
Codepython
from sklearn.metrics import roc_auc_score, average_precision_score
# Fraud detection: ~1% of samples are positive (highly imbalanced)
def evaluate(model, X_test, y_test):
y_pred = model.predict(X_test) # returns hard 0/1 labels
roc = roc_auc_score(y_test, y_pred)
pr = average_precision_score(y_test, y_pred)
return {"roc_auc": roc, "pr_auc": pr}
# We plan to compare models and later tune a decision threshold
# by cost, using these two threshold-independent scores.
The reported roc_auc and pr_auc look suspiciously low and don't change when the model improves its ranking. What is the bug?