Machine Learning�Week 10 – Deep Learning (3)
Announcement
| Week | |||||||
| W9 | W10 | W11 | W12 | W13 | W14 | W15 | W16 |
Assign#3 (project proposal) | | start | | end | | | | |
Assign#4 (data collection and analysis) | | | start | | end | | | |
Assign#5 (model training and evaluation) | | | | start | | end | | |
Assign#6 (real usage & final report) | | | | | start | | end | |
Assign#7 (presentation) | | | | | | | lecture | |
Lecture Topic | Unsup. Learning | Unsup.�Learning | Deep Learning | Deep Learning | Advanced�Topics | Advanced�Topics | Final�Project | Final Exam |
Deep Learning
Convolutional Neural Networks
What Computers “See”?
Tasks in Computer Vision
Input Image
Pixel Representation
Lincoln
Washington
Jefferson
Obama
Trump
0.8
0.05
0.05
0.01
0.09
classification
“Learning” Feature Representations
Fully Connected Neural Network
Fully Connected Layer
Locally Connected Layer
Convolutional Layer
Key Idea
The Convolution Operation
Image
Kernel
Feature Map
Feature Extraction with Convolution: A Case Study
-1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 |
-1 | 1 | -1 | -1 | -1 | -1 | -1 | 1 | -1 |
-1 | -1 | 1 | -1 | -1 | -1 | 1 | -1 | -1 |
-1 | -1 | -1 | 1 | -1 | 1 | -1 | -1 | -1 |
-1 | -1 | -1 | -1 | 1 | -1 | -1 | -1 | -1 |
-1 | -1 | -1 | 1 | -1 | 1 | -1 | -1 | -1 |
-1 | -1 | 1 | -1 | -1 | -1 | 1 | -1 | -1 |
-1 | 1 | -1 | -1 | -1 | -1 | -1 | 1 | -1 |
-1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 |
-1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 |
-1 | -1 | -1 | -1 | -1 | -1 | 1 | -1 | -1 |
-1 | 1 | -1 | -1 | -1 | 1 | -1 | -1 | -1 |
-1 | -1 | 1 | 1 | -1 | 1 | -1 | -1 | -1 |
-1 | -1 | -1 | -1 | 1 | -1 | -1 | -1 | -1 |
-1 | -1 | -1 | 1 | -1 | 1 | 1 | -1 | -1 |
-1 | -1 | -1 | 1 | -1 | -1 | -1 | 1 | -1 |
-1 | -1 | 1 | -1 | -1 | -1 | -1 | -1 | -1 |
-1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 |
Feature Extraction with Convolution: A Case Study
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
1 | -1 | -1 |
-1 | 1 | -1 |
-1 | -1 | 1 |
1 | -1 | 1 |
-1 | 1 | -1 |
1 | -1 | 1 |
-1 | -1 | 1 |
-1 | 1 | -1 |
1 | -1 | -1 |
Feature Extraction with Convolution: A Case Study
1 | X | 1 | = | 1 |
-1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 |
-1 | 1 | -1 | -1 | -1 | -1 | -1 | 1 | -1 |
-1 | -1 | 1 | -1 | -1 | -1 | 1 | -1 | -1 |
-1 | -1 | -1 | 1 | -1 | 1 | -1 | -1 | -1 |
-1 | -1 | -1 | -1 | 1 | -1 | -1 | -1 | -1 |
-1 | -1 | -1 | 1 | -1 | 1 | -1 | -1 | -1 |
-1 | -1 | 1 | -1 | -1 | -1 | 1 | -1 | -1 |
-1 | 1 | -1 | -1 | -1 | -1 | -1 | 1 | -1 |
-1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 |
1 | -1 | -1 |
-1 | 1 | -1 |
-1 | -1 | 1 |
1 | -1 | -1 |
-1 | 1 | -1 |
-1 | -1 | 1 |
= | 9 |
element-wise multiply
add outputs
Producing Feature Maps
Original
Sharpen
Edge Detect
“Strong” Edge Detect
Pooling Layer
Pooling Layer
Spatial Pooling
Convolutional Neural Networks
Important Concepts in CNN
Important Concepts in CNN
2D Conv on 3D input
3D Conv on 3D input
Important Concepts in CNN
Important Concepts in CNN
Important Concepts in CNN
Representation Learning in Deep CNNs
Conv Layer 1
Conv Layer 2
Conv Layer 3
Practice 1: Feature Map Shape
Practice 1: Feature Map Shape
Convolution with 3x3 kernel, zero padding and stride = 1
Practice 1: Feature Map Shape
Convolution with 3x3 kernel, zero padding and stride = 2
Deep Learning
Recurrent Neural Networks
So Far
Sequence Matters
???
Sequence Matters
What is a Sequence?
Sequence Modeling
given previous words
predict the next word
A Sequence Modeling Example: Next Word Prediction
“This morning I took my dog for a walk.”
given previous two words
predict the next word
A Sequence Modeling Example: Next Word Prediction
“This morning I took my dog for a walk.”
predict the next word
Here 1 is the count for the word “a”
[0 1 0 0 1 0 1 … … 0 0 1 1 0 0 0 1 0]
Sequence Modeling
A Recurrent Neural Network (RNN)
output vector
input vector
cell state
old state
current input
Standard Feed-Forward Neural Network
Recurrent Neural Networks
… and many other architectures and applications
RNN: State Update and Output
output vector
input vector
cell state
old state
current input
RNN: Computational Graph across Time
RNN: Computational Graph across Time
RNN: Computational Graph across Time
RNN: Computational Graph across Time
RNN: Computational Graph across Time
RNN: Computational Graph across Time
RNN: Computational Graph across Time
RNN: Backpropagation Through Time
Standard RNN Gradient Flow
Standard RNN Gradient Flow: Exploding Gradients
Many values > 1:
exploding gradients
Gradient clipping to�scale big gradients
Many values < 1:
vanishing gradients
Standard RNN Gradient Flow: Vanishing Gradients
Many values > 1:
exploding gradients
Gradient clipping to�scale big gradients
Many values < 1:
vanishing gradients
Why is Exploding Gradient a Problem?
Solution: Gradient Clipping
The Problem of Long-Term Dependencies
Multiply many small numbers together
Errors due to further back time steps �have smaller and smaller gradients
Bias parameters to capture short-term dependencies
Vanishing Gradient Intuition
Vanishing Gradient Intuition
Chain rule!
Vanishing Gradient Intuition
Chain rule!
Vanishing Gradient Intuition
Vanishing gradient problem:
When these are small, the gradient signal gets smaller and smaller as it backpropagates further
Vanishing Gradient Intuition
Vanishing gradient problem:
When these are small, the gradient signal gets smaller and smaller as it backpropagates further
Why is Vanishing Gradient a Problem?
Gradient signal from far away is lost because it’s much smaller than gradient signal from close-by.
So, model weights are basically updated only with respect to near effects, not long-term effects.
Why is Vanishing Gradient a Problem?
Solution: Gating Mechanisms in Neurons
Standard RNNs
Long Short-Term Memory (LSTM)
Long Short-Term Memory (LSTM)
Long Short-Term Memory (LSTM)
Gates optionally let information through, via a sigmoid layer and pointwise multiplication
LSTM: Forget Irrelevant Information
LSTM: Add New Information
LSTM: Update Cell State
LSTM: Output Filtered Version of Cell State
LSTM: Cell State Matters
LSTM: Cell State Matters
LSTM: Mitigate Vanishing Gradient
… Vanish!
… Explode!
🡪
🡪
LSTM: Mitigate Vanishing Gradient
So…
We can keep information if we want!�(by adjusting how much we forget)
LSTM: Key Concepts
Supplementary: Residual Connection
RNN Applications
RNN Applications
How to compute sentence encoding?
RNN Applications
Basic way: use final hidden state
RNN Applications
Usually better: �Take element-wise max or mean of all hidden states
RNN Applications
RNN Applications & Limitations
RNN Applications
Bidirectional and Multi-layer RNNs
We can regard this hidden state as a representation of the word “terribly” in the context of this sentence. We call this a contextual representation.
element-wise mean/max
element-wise mean/max
the
movie
was
terribly
exciting
!
These contextual representations only contain information about the left context (e.g., “the movie was”).
What about right context?
In this example, “exciting” is in the right context and this modifies the meaning of “terribly” (from negative to positive)
Bidirectional and Multi-layer RNNs
This contextual representation of “terribly” has both left and right context!
Concatenated hidden states
Forward RNN
Backward RNN
Bidirectional RNNs
This is a general notation to mean “compute one forward step of the RNN” – it could be a simple RNN or LSTM computation.
Concatenated hidden states
Forward RNN
Backward RNN
We regard this as “the hidden state” of a bidirectional RNN.
This is what we pass on the next parts of the network.
Generally, these two RNNs have separate weights
Bidirectional RNNs: Simplified Diagram
The two-way arrows indicate bidirectionality and the depicted hidden states are assumed to be the concatenated forwards+backwards states
Bidirectional RNNs
Multi-layer RNNs
Multi-layer RNNs
RNN layer 1
RNN layer 2
RNN layer 3
Multi-layer RNNs
RNN Limitations
RNN Limitations
In Summary
Next
Assignment #4