Why accuracy misleads on skewed datasets and what to measure instead.
Codepython
from sklearn.metrics import accuracy_score
# 95 negatives, 5 positives
y_true = [0]*95 + [1]*5
y_pred = [0]*100 # model predicts everything negative
score = accuracy_score(y_true, y_pred)
# We conclude the fraud detector works great because score is high
print(score)
Why is the conclusion that the detector 'works great' a bug in this evaluation?