Recurrent Networks and Sequence Data

Site Admin · 11 Sep 2026 · 9 views

Recurrent Networks and Sequence Data

Images and tables ignore order; text, audio, and sensor logs do not. Recurrent neural networks (RNNs) process sequences one step at a time and carry a hidden state forward, so the network remembers what happened earlier in the input.

Why Order Needs Its Own Machinery

In the sentence about a cat chasing a dog and the sentence about a dog chasing a cat, the words are the same but the meaning flips. A network that treats the sentence as a bag of words cannot tell the two apart. An RNN reads tokens in order and updates its hidden state after each one, so later decisions depend on earlier context.

The Hidden State

At each step, the RNN combines the current input with the previous hidden state, applies a nonlinearity, and produces a new hidden state. That state is a compressed memory. With enough steps, the memory of the first token fades, which is why plain RNNs struggle with long context.

LSTMs and GRUs Fix the Memory

Long short-term memory (LSTM) units add a separate cell state and learned gates that decide what to remember, what to write, and what to forget. Gated recurrent units (GRUs) are a leaner version. Both let gradients flow over hundreds of steps and are what people usually mean when they say RNN.

A One-Liner in PyTorch

import torch.nn as nn

rnn = nn.LSTM(input_size=10, hidden_size=32, batch_first=True)
output, (h, c) = rnn(sequence)

output holds the state at every step; h and c hold the final hidden and cell states, useful for a classifier that decides after reading the whole sequence.

Modern Alternatives

Transformers handle long sequences better than RNNs and dominate for language, but RNNs remain useful for streaming prediction, small models, and tasks with rigid ordering such as a time series tick stream. Learn the RNN machinery once, and the transformer architecture becomes an easier second step.