Backpropagation and Gradient Descent

Site Admin · 11 Sep 2026 · 8 views

Backpropagation and Gradient Descent

Gradient descent says how to improve weights; backpropagation says how to compute the gradients efficiently. Together they are the training loop that every network uses.

Backward Mode, Step by Step

The loss depends on every weight. Computing hundreds of thousands of effects independently would be absurd, so backpropagation exploits the chain rule: it carries the error backward from the output, layer by layer, reusing values. Each layer contributes to the gradient of the layers before it, and one backward pass fills every weight's slope.

Gradient Descent in Action

import torch

x = torch.tensor([3.0], requires_grad=True)
loss = (x - 5) ** 2
loss.backward()
print(x.grad)

Here the loss is minimized at x equals 5, and the gradient points toward steeper loss. Moving x against the gradient, scaled by a learning rate, walks toward the minimum.

The Learning Rate Trade-off

Too large a learning rate overshoots and diverges; too small crawls and wastes compute. Practice shows that decaying schedules and adaptive methods such as Adam tune the step per parameter automatically, which is why Adam is the common default.

Mini-Batch Training

Computing gradients over the whole dataset is exact but slow, and over one sample is fast but noisy. Mini-batches of 32 or 64 examples balance the two: enough samples for a stable gradient, small enough to fit in memory and train quickly. One pass over all batches is an epoch.

Local Minima and the Plateau

Optimization can stall in a flat region even when a better answer exists. Jitter from mini-batches often escapes for free; when it does not, raise the learning rate briefly or restart. Perfect minima matter less than good ones that generalize.