Long Short-Term Memory (LSTM)

Detailed explanation • Architecture • Gates • Worked numerical example

Input → Gates → Cell State → Hidden State

A practical teaching presentation for AI / ML students

1. What is LSTM?

Long Short-Term Memory is a specialized Recurrent Neural Network (RNN).

Definition

LSTM is designed to learn dependencies across long sequences.

It maintains a cell state (long-term memory) and a hidden state (current output).

Gates regulate what information is forgotten, added, and exposed.

Why LSTM?

Traditional RNNs can suffer from vanishing or exploding gradients.

LSTM provides a controlled memory pathway that helps preserve useful information over many time steps.

Sequential Data

Text

Speech

Time series

Video

Sensor streams

Core Idea

Remember useful information

Forget irrelevant information

Use current information to produce an output

Key States

Cell state Cₜ = long-term memory

Hidden state hₜ = short-term/current output

2

2. LSTM Cell Architecture

At each time step, the cell receives xₜ, hₜ₋₁ and Cₜ₋₁.

3

3. The Four Main Operations

The gates use sigmoid activation; the candidate memory uses tanh.

① Forget Gate fₜ

Decides how much of the previous cell state Cₜ₋₁ to keep.

​

Output range: 0 to 1

fₜ = σ(W_f xₜ + U_f hₜ₋₁ + b_f)

② Input Gate iₜ

Controls how much new information enters the cell state.

​

Output range: 0 to 1

iₜ = σ(W_i xₜ + U_i hₜ₋₁ + b_i)

③ Candidate C̃ₜ

Creates candidate information that can be added to memory.

​

Range: −1 to 1

C̃ₜ = tanh(W_c xₜ + U_c hₜ₋₁ + b_c)

④ Output Gate oₜ

Controls how much of the updated memory becomes the hidden state.

​

Output range: 0 to 1

oₜ = σ(W_o xₜ + U_o hₜ₋₁ + b_o)

Cₜ = fₜ × Cₜ₋₁ + iₜ × C̃ₜ

hₜ = oₜ × tanh(Cₜ)

4

4. How Information Flows Through LSTM

A compact view of one time step.

Input

xₜ + hₜ₋₁ + Cₜ₋₁

→

Forget

fₜ × Cₜ₋₁

→

Add

iₜ × C̃ₜ

→

Update

Cₜ = retained + new

→

Output

hₜ = oₜ × tanh(Cₜ)

Think of Cₜ as the memory highway; gates control what gets through.

5

5. Worked Numerical Example

One LSTM time step using scalar values for easy calculation.

Given

Current input: xₜ = 0.5

Previous hidden state: hₜ₋₁ = 0.1

Previous cell state: Cₜ₋₁ = 0.2

​

Forget: W_f = 0.8, U_f = 0.2, b_f = 0.1

Input: W_i = 0.5, U_i = 0.3, b_i = 0

Candidate: W_c = 0.4, U_c = 0.1, b_c = 0

Output: W_o = 0.7, U_o = 0.2, b_o = 0.1

Activation Functions

Sigmoid: σ(z) = 1 / (1 + e⁻ᶻ)

Sigmoid output lies between 0 and 1.

tanh(z) lies between −1 and 1.

Goal: compute fₜ, iₜ, C̃ₜ, Cₜ, oₜ and hₜ

All weights are scalars in this teaching example.

6

6. Step 1 — Compute the Forget Gate

The forget gate decides how much old memory is retained.

fₜ = σ(W_f xₜ + U_f hₜ₋₁ + b_f)

fₜ = σ(0.8×0.5 + 0.2×0.1 + 0.1)

fₜ = σ(0.52)

fₜ ≈ 0.627

Interpretation

About 62.7% of the previous cell-state information is retained.

7

7. Step 2 — Compute the Input Gate

The input gate controls how much new information is written.

iₜ = σ(W_i xₜ + U_i hₜ₋₁ + b_i)

iₜ = σ(0.5×0.5 + 0.3×0.1 + 0)

iₜ = σ(0.28)

iₜ ≈ 0.570

Interpretation

About 57.0% of the candidate information is allowed into memory.

8

8. Step 3 — Compute the Candidate Cell State

The candidate state proposes new information for the memory.

C̃ₜ = tanh(W_c xₜ + U_c hₜ₋₁ + b_c)

C̃ₜ = tanh(0.4×0.5 + 0.1×0.1 + 0)

C̃ₜ = tanh(0.21)

C̃ₜ ≈ 0.207

Interpretation

The candidate value is the new information that may be written to the cell state.

9

9. Step 4 — Update the Cell State

Combine retained old memory with gated new information.

Cₜ = fₜ × Cₜ₋₁ + iₜ × C̃ₜ

Cₜ = (0.627×0.2) + (0.570×0.207)

Cₜ = 0.1254 + 0.1180

Cₜ ≈ 0.2434

Meaning

The LSTM has now created its updated long-term memory.

10

10. Step 5 — Compute the Output Gate

The output gate controls how much updated memory becomes visible.

oₜ = σ(W_o xₜ + U_o hₜ₋₁ + b_o)

oₜ = σ(0.7×0.5 + 0.2×0.1 + 0.1)

oₜ = σ(0.47)

oₜ ≈ 0.615

11

11. Step 6 — Compute the Hidden State

The hidden state is the current output passed to the next time step.

hₜ = oₜ × tanh(Cₜ)

hₜ = 0.615 × tanh(0.2434)

tanh(0.2434) ≈ 0.239

hₜ = 0.615 × 0.239

hₜ ≈ 0.147

12

12. Complete Calculation — Summary

All intermediate values from the worked example.

Final: Cₜ ≈ 0.2434 and hₜ ≈ 0.147

13

13. Why LSTM Works for Long-Term Dependencies

Memory Pathway

The cell state carries information forward through time.

​

This provides a relatively direct path for important information.

Controlled Updates

Sigmoid gates act like soft switches.

​

They learn what to keep, add, and output.

Long Sequences

Useful for dependencies that may span many time steps.

​

Common in text, speech, forecasting and other sequential tasks.

• Captures long-term dependencies

• Reduces the vanishing-gradient difficulty of basic RNNs

• Works with variable-length sequences

• Can be stacked or combined with other neural-network layers

14

14. Applications of LSTM

Natural Language Processing

• Language modeling

• Text generation

• Sentiment analysis

• Sequence classification

• Machine translation (historically common)

Speech & Signals

• Speech recognition

• Audio sequence modeling

• Sensor data

• Human activity recognition

• Anomaly detection

Time-Series & Video

• Demand forecasting

• Stock / financial sequences

• Weather-related sequences

• Video sequence analysis

• Predictive maintenance

15

15. Final Takeaway

Remember the LSTM as a sequence of controlled memory decisions.

FORGET → ADD → UPDATE MEMORY → OUTPUT

• Forget gate fₜ: decides what old information to remove or retain.

• Input gate iₜ: decides how much new information to write.

• Candidate C̃ₜ: proposes new content for memory.

• Cell state Cₜ: stores the updated long-term information.

• Output gate oₜ: decides what part of memory becomes the hidden state.

• Hidden state hₜ: the output passed to the next time step.

Worked example result: Cₜ ≈ 0.2434, hₜ ≈ 0.147

16

SEQUENCE MODELS

Gated Recurrent

Unit (GRU)

A simpler, faster gated cell for sequence learning — architecture, equations, and a worked example.

Reset Gate

Update Gate

Fewer Params

AGENDA

What we'll cover

Why GRU exists

A quick recap of the vanishing-gradient problem and why a simpler gated cell is useful.

GRU architecture

The reset gate, update gate, and candidate state — laid out as a full diagram.

The equations

Every gate's formula explained, plus a fully worked numeric example.

GRU vs LSTM vs RNN

How it stacks up, and when to reach for it instead of the alternatives.

Gated Recurrent Unit (GRU)

2

MOTIVATION

Why GRU? A Simpler Fix for Long Memory

Vanilla RNNs forget quickly. LSTMs fix that with three gates and a separate cell state. GRU asks: can we get most of the benefit with less machinery?

Vanilla RNN

One hidden state, no gates. Fast, but gradients vanish over long sequences — early context gets forgotten.

LSTM

Adds a separate cell state plus 3 gates (forget, input, output). Excellent memory, but more parameters and slower to train.

GRU

Merges the cell state into the hidden state and uses just 2 gates (reset, update). Nearly LSTM-level memory with a leaner design.

The result: GRU trains faster and needs less data than LSTM, while still solving the vanishing-gradient problem that limits vanilla RNNs — making it a strong default choice for many sequence tasks.

Gated Recurrent Unit (GRU)

3

ARCHITECTURE

Inside the GRU Cell

A single hidden-state line runs through the cell. Two gates decide how much of the past to keep and how much new information to blend in.

h_{t−1}

hₜ

Reset Gate

rₜ = σ

Update Gate

zₜ = σ

Candidate

h̃ₜ = tanh

rₜ ⊙ h_{t−1}

xₜ

Σ

zₜ

h̃ₜ

(pass-through h_{t−1}, scaled by 1−zₜ, blends in at Σ)

hₜ = (1 − zₜ) ⊙ h_{t−1} + zₜ ⊙ h̃ₜ

No separate cell state — the update gate zₜ directly blends old and new hidden state on one shared line.

Gated Recurrent Unit (GRU)

4

THE EQUATIONS

The Three GRU Equations

Everything a GRU does comes down to three formulas, computed in order at every time step.

1

Reset gate

rₜ = σ(Wr · [h_{t−1}, xₜ] + br)

Decides how much of the previous hidden state to forget before computing the candidate. rₜ ≈ 0 means "ignore the past here."

2

Update gate

zₜ = σ(Wz · [h_{t−1}, xₜ] + bz)

Decides the blend ratio between old and new state. zₜ ≈ 1 means "replace the memory with new information."

3

Candidate state

h̃ₜ = tanh(W · [rₜ ⊙ h_{t−1}, xₜ] + b)

A proposed new hidden state, computed using only the reset-gated portion of history plus the current input.

4

New hidden state

hₜ = (1 − zₜ) ⊙ h_{t−1} + zₜ ⊙ h̃ₜ

The final output — a weighted average of the old state and the candidate, controlled entirely by zₜ.

Gated Recurrent Unit (GRU)

5

WORKED EXAMPLE

A GRU Step, Computed by Hand

Reading the word "amazing" right after a lukewarm sentence — watch how the gates decide what to keep.

Reset gate

rₜ = σ(Wr·[h_{t−1},xₜ]+br)

rₜ ≈ 0.20

Mostly reset — the old context barely informs the candidate

Update gate

zₜ = σ(Wz·[h_{t−1},xₜ]+bz)

zₜ ≈ 0.85

Lean heavily toward the new information

Gated history

rₜ ⊙ h_{t−1}

0.20 × 0.10 = 0.02

Almost nothing of the old state reaches the candidate

Candidate state

h̃ₜ = tanh(W·[rₜ⊙h_{t−1},xₜ]+b)

h̃ₜ ≈ 0.90

"Amazing" drives a strongly positive candidate

New hidden state

hₜ = (1−zₜ)·h_{t−1} + zₜ·h̃ₜ

0.15×0.10 + 0.85×0.90 ≈ 0.78

The state jumps sharply positive, dominated by the new input

Notice: the reset gate suppressed old context while the candidate formed, and the update gate then let that fresh, positive candidate dominate the new hidden state — all with two gates instead of LSTM's three.

Gated Recurrent Unit (GRU)

6

GRU VS LSTM

GRU vs LSTM: Same Idea, Less Machinery

Both use gates to control memory. GRU gets there with one state and two gates instead of two states and three.

LSTM

2 states: cell state Cₜ + hidden state hₜ

3 gates: forget, input, output

Candidate gated separately, then added to cell state

More parameters — roughly 4× per unit vs. a plain RNN

Slightly stronger on very long, complex dependencies

GRU

1 state: just the hidden state hₜ

2 gates: reset, update

Candidate directly blended into the hidden state via zₜ

Fewer parameters — roughly 25% fewer than LSTM

Trains faster, often matches LSTM on shorter/medium sequences

Gated Recurrent Unit (GRU)

7

COMPARISON

RNN vs GRU vs LSTM

Vanilla RNN

GRU

LSTM

Memory

Single hidden state

Single hidden state

Hidden + cell state

Gates

None

Reset, Update

Forget, Input, Output

Long-range memory

Weak

Strong

Strong

Parameters per unit

Fewest

~25% fewer than LSTM

Most

Training speed

Fastest

Fast

Slower

Good default when...

Sequences are short

You want strong results with fewer params

You need maximum capacity on long, complex sequences

Gated Recurrent Unit (GRU)

8

USE CASES

Where GRU Shows Up in Practice

Its speed and lower data requirements make GRU a common default before reaching for something heavier.

Speech recognition

Streaming audio → text, where inference speed matters as much as accuracy.

Sentiment & text classification

Reviews, tweets, support tickets — sequences short enough that GRU's memory is plenty.

Time-series forecasting

Sensor data, traffic, energy demand — fast training over rolling windows.

Music & handwriting generation

Sequential generation tasks where a lighter cell speeds up sampling.

On-device / mobile models

Fewer parameters than LSTM means a smaller memory and compute footprint.

Baseline before LSTM

A fast, strong first attempt — many teams only move to LSTM if GRU underperforms.

Gated Recurrent Unit (GRU)

9

SUMMARY

Key Takeaways

1

Two gates, one state

Reset (rₜ) controls how much history reaches the candidate; update (zₜ) blends old and new directly on the hidden-state line.

2

No separate cell state

Unlike LSTM, GRU has nothing beyond hₜ — simpler bookkeeping, fewer parameters to learn.

3

Fixes vanishing gradients

The (1−zₜ) pass-through path lets gradients flow across many time steps, just like LSTM's cell-state highway.

4

A strong, fast default

Often matches LSTM accuracy with less data and less compute — a sensible first choice before reaching for something bigger.