Overfitting, Dropout and Regularization

Site Admin · 11 Sep 2026 · 8 views

Overfitting, Dropout and Regularization

An overfit model memorizes its training data and fails on the next batch of real data. The symptom is diagnostic: the training loss keeps falling while the validation loss flattens or climbs. Overfitting is the most common failure mode in deep learning.

The Root of the Problem

Networks have far more parameters than examples. Given enough capacity and time, a network can encode the exact labels instead of the general pattern. The cure is never one trick; it is a stack of them, each reducing how easily the model copies the training set.

Dropout Breaks Co-Adaptation

Dropout randomly turns off a fraction of neurons on every forward pass. The network can no longer rely on any single neuron, so it spreads knowledge across the network. At inference the neurons all run, scaled to keep the layer's output stable.

import torch.nn as nn

self.fc1 = nn.Linear(128, 64)
self.drop = nn.Dropout(0.5)
self.fc2 = nn.Linear(64, 10)

def forward(self, x):
    x = torch.relu(self.fc1(x))
    return self.fc2(self.drop(x))

Rates near 0.5 are common for dense layers; a smaller rate, near 0.1, fits convolutional layers and mild problems.

Weight Decay and Early Stopping

Weight decay (L2 regularization) penalizes large weights, quietly keeping the function simple. Early stopping watches validation loss and keeps the checkpoint from before it started rising. Both are nearly free in PyTorch and belong in every serious run.

Data-Side Regularizers

More data is the strongest regularizer of all. When the dataset is fixed, augmentation manufactures variations: rotating, flipping, and shifting images, or permuting tokens. Each variation forces the model to learn invariants instead of pixels.

Follow the Validation Curve

Plot train and validation loss on the same chart. A growing gap is overfitting; apply dropout or weight decay and rerun. One change per run keeps the diagnosis honest.