SEQUENCE MODELS
Recurrent Neural
Networks (RNN)
How networks remember — architectures, math, and diagrams for RNNs, LSTMs and GRUs, with worked examples.
Vanilla RNN
LSTM
GRU
BPTT
AGENDA
What we'll cover
The RNN cell
Recurrence, unrolling through time, and the hidden-state equations.
Architectures & examples
One-to-many, many-to-one, many-to-many — with real use cases.
Training & its limits
Backpropagation through time and the vanishing gradient problem.
LSTM & GRU
Gated cells that fix long-range memory, plus a worked numeric example.
Recurrent Neural Networks
2
MOTIVATION
Why Sequences Need a Different Network
Feedforward networks and CNNs treat each input independently. Sequence data — text, speech, time series — has order, and meaning depends on what came before.
Feedforward network
No memory of previous inputs — "The movie was not good" and "not the movie was good" could look identical if word order is ignored.
Recurrent network
Carries a hidden state forward — each new input is interpreted in the context of everything read so far.
Language
Predicting the next word from previous words
Speech
Converting an audio waveform into text
Time series
Forecasting tomorrow's stock price or weather
Video
Classifying an action across many frames
Recurrent Neural Networks
3
THE RNN CELL
Recurrence: Folded vs Unrolled
The same cell and the same weights are reused at every time step — the loop on the left is just a compact way to draw the chain on the right.
Folded (compact) view
RNN Cell
h_t = tanh(Wx·xₜ + Wh·h_{t−1} + b)
xₜ
hₜ
Loop feeds the hidden state
back in at the next step
Unrolled through time
RNN
t = 1
x1
y1
RNN
t = 2
x2
y2
RNN
t = 3
x3
y3
h₀
The orange arrows carry the hidden state hₜ — the network's memory — from one step to the next. Each step can also emit its own output yₜ.
Recurrent Neural Networks
4
THE MATH
The Hidden State Update
At every step the cell combines the new input with its memory of the past, then squashes the result with tanh.
hₜ = tanh( Wxh · xₜ + Whh · h_{t−1} + b )
yₜ = softmax( Why · hₜ + c ) — an optional output produced from the hidden state at each step
Worked example — one time step
Input xₜ
[1.0, 0.0] (one-hot for the word "cat")
Previous hidden state h_{t−1}
[0.20, −0.10]
Weights (illustrative)
Wxh = [[0.5, −0.3]], Whh = [[0.1, 0.4], [−0.2, 0.3]], b = 0
Weighted sum Wxh·xₜ + Whh·h_{t−1}
[0.5×1.0 + 0.1×0.20 + (−0.2)×(−0.10), −0.3×1.0 + 0.4×0.20 + 0.3×(−0.10)] = [0.54, −0.25]
New hidden state hₜ = tanh(…)
tanh([0.54, −0.25]) ≈ [0.49, −0.24]
This new hidden state hₜ now carries forward everything the network has "read" so far, ready for the next word.
Recurrent Neural Networks
5
ARCHITECTURES
Four Ways to Wire an RNN
The same cell can be arranged to fit how many inputs and outputs a task actually has.
One → One
Image classification (not truly recurrent — shown for contrast)
One → Many
Image captioning — one image in, a sentence out
Many → One
Sentiment analysis — a review in, one label out
Many → Many
Machine translation — a sentence in, a sentence out
↑ stub = produces an output ↓ stub = receives an input
Recurrent Neural Networks
6
TRAINING
Backpropagation Through Time (BPTT)
To train an RNN, unroll it across all time steps, then run ordinary backpropagation on the unrolled graph — gradients flow backward through every copy of the cell.
RNN
t = 1
L1
RNN
t = 2
L2
RNN
t = 3
L3
RNN
t = 4
L4
Gradients from every loss L₁…L₄ flow backward (red) through the recurrent connections, all the way to the first step.
The catch: the gradient for an early time step is a product of many terms — one per step it passes back through. If those terms are consistently small, the product shrinks toward zero; if they're consistently large, it explodes. That's the vanishing/exploding gradient problem, covered next.
Recurrent Neural Networks
7
TRAINING
The Vanishing Gradient Problem
Each backward step multiplies the gradient by a factor usually well under 1 (from tanh and the recurrent weight). Over many steps, that product collapses toward zero.
Gradient magnitude reaching each earlier step (illustrative)
1.00
Step T
0.55
T−1
0.30
T−2
0.16
T−3
0.08
T−4
0.04
T−5
Why it happens
Backprop multiplies many small (<1) numbers together — tanh's derivative is ≤ 0.25, and repeated recurrent-weight multiplication compounds the shrinkage.
The consequence
Early time steps get almost no learning signal — the network effectively "forgets" long-range dependencies.
The fix
Gated architectures like LSTM and GRU (next slides) add a path where gradients flow with much less decay.
Recurrent Neural Networks
8
LSTM
LSTM: A Cell That Can Choose to Remember
A separate "cell state" runs across the top with only light, gated interactions — that's the highway gradients use to survive many time steps.
Cₜ₋₁
Cₜ
Forget Gate
σ
Input Gate
σ
Candidate
tanh
Output Gate
σ
×
×
+
tanh
×
h_{t−1}
hₜ
[h_{t−1}, xₜ] (concatenated) feeds every gate from below
Recurrent Neural Networks
9
LSTM · WORKED EXAMPLE
LSTM Gate Equations, Step by Step
Reading the word "excellent" after a slightly negative sentence — the gates decide how much of the old memory to keep vs. overwrite.
Forget gate
fₜ = σ(Wf·[h_{t−1},xₜ]+bf)
fₜ ≈ 0.30
Mostly forget the old sentiment — it was weak anyway
Input gate
iₜ = σ(Wi·[h_{t−1},xₜ]+bi)
iₜ ≈ 0.90
Let a lot of new information in
Candidate
C̃ₜ = tanh(Wc·[h_{t−1},xₜ]+bc)
C̃ₜ ≈ 0.85
"Excellent" produces a strongly positive candidate value
New cell state
Cₜ = fₜ·C_{t−1} + iₜ·C̃ₜ
Cₜ ≈ 0.30×0.10 + 0.90×0.85 ≈ 0.80
Memory shifts sharply positive
Output gate
oₜ = σ(Wo·[h_{t−1},xₜ]+bo)
oₜ ≈ 0.75
Expose most of that new state as the output
New hidden state
hₜ = oₜ · tanh(Cₜ)
hₜ ≈ 0.75 × tanh(0.80) ≈ 0.50
A confidently positive hidden state moves forward
Recurrent Neural Networks
10
GRU
GRU: A Simpler Gated Cell
The Gated Recurrent Unit merges the cell state into the hidden state and uses just two gates — fewer parameters, similar performance on many tasks.
Single state line — h_{t−1} → hₜ
h_{t−1}
hₜ
Reset Gate
σ
Update Gate
σ
Candidate
tanh
z
hₜ = (1−zₜ)·h_{t−1} + zₜ·h̃ₜ
The reset gate controls how much past state feeds the candidate; the update gate then blends old vs. new directly on the hidden-state line — no separate cell state needed.
LSTM vs GRU at a glance
LSTM
GRU
Gates
3 (forget, input, output)
2 (reset, update)
State
Separate cell + hidden state
Single hidden state
Parameters
More
~25% fewer
Typical use
Longer sequences, more capacity
Faster training, smaller datasets
Recurrent Neural Networks
11
WORKED EXAMPLE
Many → One: Sentiment Analysis Walkthrough
"the movie was not good" — the network reads one word at a time, updating a running sentiment score (hidden state) as it goes.
RNN
"the"
h=0.00
RNN
"movie"
h=0.02
RNN
"was"
h=0.05
RNN
"not"
h=−0.10
RNN
"good"
h=−0.55
Watch the hidden state: it drifts near zero through "the movie was", then "not" nudges it negative, and "good" — which alone would be positive — gets reinterpreted in context, pushing the score further negative. That's exactly what a bag-of-words model would miss.
Final classification
y = softmax(Why · h₅ + c) → "Negative" (0.87)
Recurrent Neural Networks
12
COMPARISON
RNN vs LSTM vs GRU
Vanilla RNN
LSTM
GRU
Memory
Single hidden state
Hidden + cell state
Single hidden state
Gates
None
Forget, Input, Output
Reset, Update
Long-range memory
Weak (vanishing gradient)
Strong
Strong
Parameters per unit
Fewest
Most (~4×)
Fewer (~3×)
Training speed
Fastest
Slower
Faster than LSTM
When to reach for it
Short sequences, simple baselines
Long documents, complex dependencies
Good default — strong results, fewer params
Recurrent Neural Networks
13
SUMMARY
Key Takeaways
1
Recurrence = memory
A hidden state is carried forward and combined with each new input, so order and context matter.
2
Unrolling reveals the training graph
The same weights are reused at every step; BPTT backpropagates through the whole unrolled chain.
3
Vanishing gradients limit vanilla RNNs
Long chains of small multiplications erase the learning signal for early time steps.
4
Gates fix that
LSTM and GRU add gated paths — a cell state highway or a direct hidden-state blend — that preserve gradients over long sequences.