Evaluating Model Performance Honestly

Site Admin · 11 Sep 2026 · 9 views

Evaluating Model Performance Honestly

Model evaluation is where good projects separate from demos. A single number never tells the whole story. Collect a confusion matrix, the class-level metrics, and a check against a baseline before you trust any score.

The Confusion Matrix

from sklearn.metrics import confusion_matrix, classification_report

preds = clf.predict(X_test)
print(confusion_matrix(y_test, preds))
print(classification_report(y_test, preds))

The matrix counts true positives, true negatives, false positives, and false negatives. The report adds precision, recall, and F1 per class. Read the per-class rows, not just the accuracy line.

Precision, Recall, and F1

Precision: of the things you labeled positive, how many were right?
Recall: of the actual positives, how many did you catch?
F1: the harmonic mean, useful when both matter equally.

Which one matters depends on the cost of each mistake. Missing a cancer is worse than a false alarm, so recall matters. Marking good email as spam is too costly, so precision matters. Always tie the metric to the business cost.

AUC and ROC for Ranking

The ROC curve plots recall against the false positive rate across thresholds; AUC summarizes it as one number. An AUC of 0.5 means random guessing, and 1.0 means perfect ranking. It is the right tool when the decision threshold is not fixed yet.

Always Compare to a Baseline

from sklearn.dummy import DummyClassifier

base = DummyClassifier(strategy="most_frequent")
base.fit(X_train, y_train)
print(base.score(X_test, y_test))

A model must beat a dumb baseline before it is worth deploying. If your fancy model barely outperforms always-predict-the-majority, the problem is not the model, it is the data.