Loss Functions and Optimizers

Site Admin · 11 Sep 2026 · 7 views

Loss Functions and Optimizers

The loss function defines what good means; the optimizer defines how to reach it. Every training configuration you will see is a choice of these two pieces plus the model that sits between them.

Regression Losses

Mean squared error (MSE) punishes large errors heavily, which suits smooth targets but overreacts to outliers. Mean absolute error (MAE) is steady against outliers. When neither fits, Huber loss combines both: quadratic near zero, linear far out.

Classification Losses

Cross-entropy rewards confident correct answers and punishes confident wrong ones, acting as the natural partner of softmax for classification. Binary cross-entropy is the two-class version. Raw logits, not softened probabilities, are what these losses expect from the network.

Optimizers: SGD and Adam

Plain SGD steps each weight by the gradient times a learning rate; it is simple, predictable, and a great baseline. Adam adapts the step per parameter using estimates of gradient moments, so it trains fast and needs little tuning at the cost of slightly messier optimization behavior. Momentum-based variants sit in between.

import torch.optim as optim

sgd = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
adam = optim.Adam(model.parameters(), lr=0.001)

Learning Rate Schedules

Training typically benefits from a schedule: a high rate to make progress, then decay so the last steps fine-tune. PyTorch provides schedulers such as step decay and cosine annealing off the shelf:

sched = optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.5)

Call sched.step() after each epoch.

Diagnose, Then Fix

If the loss does not move, the learning rate is probably too small or the inputs are unscaled. If it explodes to NaN, lower the rate or shrink the model. Record the loss every epoch; the curve is the fastest diagnostic you have.