Train/Test Splits: Trustworthy Evaluations
Train/Test Splits: Trustworthy Evaluations
You evaluate a model, but the real question is how it will perform on data it has never seen. If you score a model on the same data used to train it, the result is inflated and lies. A train/test split answers the honest version of the question.
Split the Data
from sklearn.model_selection import train_test_split
X = data.drop("price", axis=1)
y = data["price"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)test_size=0.2 holds out 20 percent. random_state freezes the shuffle so every run of your script produces the identical split. Fix it once you have a seed you like; never tune models against the test set.
Why Order and Shuffling Matter
Data often arrives sorted by time or by some key. A naive first-80-percent split then copies that structure into the model. Shuffle before splitting so both sets cover the full range of examples.
Three Sets Are Better Than Two
Add a validation set for comparing models and tuning parameters; keep the test set for a single final check. Shape the split as train, validation, and test, for example 70-15-15. Tuning on the test set is the classic self-deception that yields a great score and a production failure.
Watch for Leakage
Any step that looks at statistics of the whole dataset can leak information. Scale the train set and apply its fitted scaler to the test set, do the same for imputation, and handle categorical levels consistently. Leakage is silent and it flatters every result.
Key Points
- Test set must stay unseen until the final check.
- Fix the random seed for reproducible splits.
- Shuffle to avoid copying data structure.
- Fit preprocessing on train, apply to test.
