Decision Trees and Random Forests

Site Admin · 11 Sep 2026 · 7 views

Decision Trees and Random Forests

A decision tree splits the data with a series of if-else questions on the features until each branch holds mostly one class or one value. Trees are easy to explain, need little preprocessing, and ignore the scaling problems that plague linear models.

How a Tree Grows

At every node the algorithm chooses the feature and threshold that split the examples into the purest groups. It repeats until leaves are small or pure. The price of this flexibility is variance: a tree grown on slightly different data can look completely different.

A Single Tree Is Too Loud

A single deep tree memorizes noise in training data. The standard fix is the random forest: many trees, each built on a random sample of rows and a random subset of features, whose votes are averaged for classification or regression. The averaging smooths away individual errors.

Train a Random Forest

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(
    n_estimators=200, max_depth=8, random_state=42
)
rf.fit(X_train, y_train)
print(rf.score(X_test, y_test))

n_estimators is the number of trees, and max_depth caps tree size and fights overfitting. More trees cost time; a depth cap usually buys more accuracy than raising the tree count.

Feature Importance

Forests report which features mattered most on average:

for name, imp in zip(rf.feature_names_in_, rf.feature_importances_):
    print(name, round(imp, 3))

Importances help you prune irrelevant columns, but treat them as guidance, not law: correlated features share credit.

When to Reach for Forests

Choose forests for mixed numeric and categorical data, nonlinear patterns, and small-to-medium datasets. They rarely need feature scaling and are a strong default before you reach for deep learning.