Recurrent Neural Networks

Learning Objectives

• ✓Map sequence problems to one-to-one, one-to-many, many-to-one, or many-to-many layouts
• ✓Compute the hidden-state recurrence of a vanilla RNN
• ✓Explain vanishing and exploding gradients through time
• ✓Trace the forget, input, cell-update, and output operations of an LSTM
• ✓Describe how a CNN encoder and recurrent decoder produce an image caption

Why Sequences Need State

A feed-forward network treats each input independently. A recurrent network maintains an internal state that is updated as a sequence is processed, so the prediction at time $t$ can depend on earlier inputs. The same parameters are reused at every timestep, which lets one model process variable-length input without growing its parameter count. The price is sequential computation and a finite hidden state that can struggle to preserve information from the distant past.

The Vanilla RNN Recurrence

Unrolling an RNN draws one copy of the recurrent cell per timestep, but all copies share $W_{xh}$, $W_{hh}$, and the biases. The current input and previous hidden state produce the new hidden state; a separate projection produces the output. During teacher-forced sequence generation, the decoder receives the correct previous token while training. During inference, it feeds back its own generated token, so mistakes can compound.

Four Input-Output Patterns

One-to-one is ordinary classification. One-to-many maps a single input to a sequence, as in caption generation. Many-to-one consumes a sequence and predicts one label, as in sentiment or video classification. Many-to-many can align outputs with inputs for tagging, or use different input and output lengths for translation and summarization. State exactly when the loss is applied: only at the last step, at every aligned step, or across the decoder sequence.

Backpropagation Through Time

BPTT applies ordinary backpropagation to the unrolled computation graph. A gradient traveling many steps contains repeated products of recurrent Jacobians. If their magnitudes are mostly below one, the gradient decays geometrically; if above one, it can explode. Vanishing gradients make long-range dependencies learn very slowly, while exploding gradients cause unstable updates. Gradient clipping addresses explosion, but preserving a usable information path requires architectural help such as gated recurrence or residual connections.

LSTM: A Gated Memory Path

An LSTM separates a cell state $C_t$ from the exposed hidden state $h_t$. The forget gate chooses how much old memory survives. The input gate controls which candidate values enter. The cell update combines retained memory and new content. The output gate decides which part of the cell becomes visible. Sigmoid gates range from zero to one, so the model can learn near-identity memory updates. This makes long dependencies easier to learn but does not mathematically guarantee that every gradient problem disappears.

GRU, Bidirectional & Stacked RNNs

A GRU combines memory and hidden state and uses only update and reset gates, reducing parameters while retaining a gated path. A bidirectional RNN processes the sequence in both directions and concatenates the states, which is appropriate only when the full input is available; it cannot be used unchanged for strictly causal streaming. Stacking recurrent layers increases capacity, but deep stacks may need dropout, normalization, or skip connections. Choose LSTM versus GRU empirically rather than assuming one always wins.

Case Study: Image Captioning

A captioner combines modalities. A pretrained CNN encodes an image into visual features. A learned projection initializes or conditions the recurrent decoder. Generation begins with a start token; each step predicts a distribution for the next word, feeds a selected word embedding back into the decoder, and stops at the end token. Greedy decoding chooses the best token each time, while beam search preserves several partial captions. The fixed-vector bottleneck motivates the visual attention mechanism introduced in the next session.

import torch
from torch import nn

embedding = nn.Embedding(vocab_size, 256)
decoder = nn.LSTM(
    input_size=256,
    hidden_size=512,
    num_layers=1,
    batch_first=True,
)
classifier = nn.Linear(512, vocab_size)

token_vectors = embedding(token_ids)      # [batch, time, 256]
states, (h_n, c_n) = decoder(token_vectors)
next_token_logits = classifier(states)   # [batch, time, vocab]