Activation Functions Explained
Activation Functions Explained
Without nonlinear activation functions, stacking layers would collapse into one giant linear layer, and the network could never learn curves or interactions. Activations are the nonlinearity that makes depth meaningful, and choosing one often decides how fast a model trains.
Why a Nonlinearity Is Required
Compose any number of linear functions and the result is still linear: no curved boundaries, no feature interactions. Insert a nonlinearity between layers and each layer can represent something the previous one could not.
ReLU: The Default
ReLU returns the input when positive and zero otherwise. It is cheap and fast to train and is more or less the default in modern models. Its weakness is dying neurons: negative inputs produce a constant zero gradient, so some weights stop learning.
import torch
import torch.nn.functional as F
x = torch.tensor([-2.0, -0.5, 0.0, 1.0, 5.0])
print("ReLU:", F.relu(x))
print("Sigmoid:", torch.sigmoid(x))
print("Tanh:", torch.tanh(x))Sigmoid and Tanh
Sigmoid squeezes values into the range zero to one, which suits probabilities and gating. Tanh centers outputs near zero, which improves gradient flow, but both saturate at the extremes, where gradients shrink toward zero and training crawls. These are best kept for the output layer of binary problems or for recurrent gates.
Softmax for Probabilities
Softmax turns raw scores into a probability distribution over classes: positive numbers that sum to one. It belongs in the final layer of a classification network, paired with a cross-entropy loss.
Match the Activation to the Role
- Hidden layers: ReLU or a variant like LeakyReLU.
- Binary output: sigmoid.
- Multi-class output: softmax.
- Recurrent layers: hyperbolic tangent or gated variants.
Key Points
- Activations add nonlinearity, the source of depth.
- ReLU is the default hidden-layer choice.
- Sigmoid and tanh fit final layers and gates, not depth.
- Softmax gives class probabilities.
