AI & ML: Evaluating Models

Site Admin · 11 Sep 2026 · 10 views

Never Grade on the Training Data

A model that memorises its training data achieves perfect scores at home and fails in the world. That is why evaluation happens on a test set, a portion of data held out before training begins. The training set teaches the model, and the test set measures how well the lesson generalises to new examples.

# split: 80% for training, 20% for testing
X_train, X_test, y_train, y_test = \
    train_test_split(X, y, test_size=0.2, random_state=42)

random_state makes the split reproducible so every experiment is comparable. Without it, results change on every run and you can never tell whether an improvement is real.

Metrics for Classification

The confusion matrix compares predictions to actual labels in a neat table.

True positive - predicted positive and actually positive.
True negative - predicted negative and actually negative.
False positive - predicted positive but actually negative.
False negative - predicted negative but actually positive.

Accuracy is the fraction of correct predictions over all predictions. Precision asks how many of the positive predictions were right. Recall asks how many of the real positives the model managed to find. A spam filter errs towards recall so it rarely misses junk, while a medical screening tool manages the same trade-off under different costs.

accuracy = correct / total
precision = true_positive / (true_positive + false_positive)
recall = true_positive / (true_positive + false_negative)

The F1 score combines precision and recall into a single number, useful when one metric alone hides a bad model.

Metrics for Regression

Regression models are judged by error statistics such as mean absolute error, which reports how far off predictions are on average, and root mean squared error, which also punishes big mistakes. Lower means better, but a score only means something relative to the target scale: an error of five is tiny for house prices and huge for body temperature.

Cross Validation

For small datasets, one fixed test split can be unlucky. Cross validation rotates the testing role across several folds and averages the results, giving a more stable estimate of quality.