Data Cleaning and Preprocessing
Data Cleaning and Preprocessing
Real data is messy: missing values, wrong types, inconsistent text, and wildly different scales. Models do not forgive mess; they simply learn the mess. Cleaning and preprocessing are where the quality of a model is decided.
Inspect for Missing Values
import pandas as pd
df = pd.read_csv("housing.csv")
print(df.isna().sum())
print(df.dtypes)Missing values arrive as NaN while missing text often arrives as an empty string, which dtypes will not flag as missing. Normalize empties to NaN first so every downstream step sees the same marker.
df = df.replace("", pd.NA)
df["price"] = pd.to_numeric(df["price"], errors="coerce")errors="coerce" turns unparseable text into NaN instead of raising; you can count the failures afterwards. Decide per column: drop the row when entry matters, impute with a median when losing the row hurts.
Encode Categories
Models work with numbers. One-hot encoding turns each category into a column of zeros and ones. scikit-learn provides OneHotEncoder, and pandas provides get_dummies for quick work:
df = pd.get_dummies(df, columns=["neighborhood"])Scale Numeric Features
Features on very different scales confuse distance-based and gradient-based models. StandardScaler subtracts the mean and divides by the standard deviation. Fit the scaler on training data only, then transform both train and test with the same scaler.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)Split the Work
Keep a CSV of the raw data untouched. Make cleaning reproducible with a script that runs end to end, and pass the cleaned set downstream. You will rerun this pipeline many times.
Key Points
- Standardize missing values before anything else.
- Encode categories into numbers.
- Scale features and fit once on training data.
- Keep cleaning in a reproducible script.
