How Neural Networks Learn
How Neural Networks Learn
A network starts as a bag of random numbers and ends, after training, as a tuned function. The mechanism is simple in outline: run data forward, compare with the answer, and nudge every weight a little in the direction that lowers the error.
The Forward Pass
Inputs enter the network, each neuron computes a weighted sum plus a bias, and a nonlinearity shapes the value before passing it to the next layer. The final layer produces the prediction. Nothing learns here; this is just using the current parameters on one batch of data.
The Error Signal
The prediction is compared with the correct answer using a loss function such as mean squared error or cross-entropy. The loss is a single number expressing how wrong the network was on this batch. Small loss means close predictions.
Weight Updates
Gradient descent repeats: compute how each weight affects the loss, then move weights slightly opposite to that effect. The step size is the learning rate. Two parameters carry the whole loop:
import torch
w = torch.tensor(1.0, requires_grad=True)
loss = (w * 4 - 8) ** 2
loss.backward()
print(w.grad)PyTorch fills w.grad with the weight's effect on the loss; gradient descent then moves w in the opposite direction and repeats.
Why Depth Matters
A single layer can only express linear boundaries. Nonlinear activations plus layers let the network fold space, compose features, and represent curved decision boundaries. Depth is not decoration; it is the source of expressive power.
What Learning Really Is
At the end, learning is thousands of tiny parameter adjustments driven by the gradient of the loss. The network stores nothing you could read; it only stores numbers that happened to work on the data it saw.
Key Points
- Forward pass computes predictions; loss measures error.
- Gradients show how each weight affects loss.
- Gradient descent nudges weights against the gradient.
- Depth plus nonlinearity provides expressive power.
