1
Institute for Data Science & Artificial Intelligence
Lecture A-2
Course: DSAI 525-Time Series and Forecasting with Machine Learning
Instructor: Ercan Atam
RNNs
List of contents for this lecture
2
Relevant readings/watching for this lecture
3
From feed-forward ANNs to RNNs
Recurrent Neural Networks (RNNs)
connections (loops).
4
Unit delay
RNNs
Examples of applications for sequence models
5
The goal of RNNs
6
Used convention & notation for RNNs (1)
7
Unit delay
RNNs
Used convention & notation for RNNs (2)
8
Unit delay
RNNs
Used convention & notation for RNNs (3)
9
Unit delay
RNNs
Unfolded RNN block diagram
10
Q: Can we show “mathematically” the equivalence?
(b) Unfolded RNNs
…
Unit delay
(a) RNNs
Mathematically equivalence of the two diagrams (1)
11
(1)
Unit delay
(a) RNNs
Mathematically equivalence of the two diagrams (2)
12
Thus, (1) and (2) are the same for any given time point t.
(b) Unfolded RNNs
…
(2)
How do we train RNNs? 🡪 Backpropagation through time (1)
13
…
How do we train RNNs? 🡪 Backpropagation through time (2)
14
Consider the i-th sample and for simplicity of notation, omit the superscript i in the figure.
…
How do we train RNNs? 🡪 Backpropagation through time (3)
15
How do we train RNNs? 🡪 Backpropagation through time (4)
16
Unit delay
How do we train RNNs? 🡪 Backpropagation through time (5)
17
Unit delay
How do we train RNNs? 🡪 Backpropagation through time (6)
18
How do we train RNNs? 🡪 Backpropagation through time (7)
19
…
How do we train RNNs? 🡪 Backpropagation through time (8)
20
How do we train RNNs? 🡪 Backpropagation through time (9)
21
How do we train RNNs? 🡪 Backpropagation through time (10)
22
Unit delay
How do we train RNNs? 🡪 Backpropagation through time (11)
23
Unit delay
(14)
(15)
(16)
How do we train RNNs? 🡪 Backpropagation through time (12)
24
How do we train RNNs? 🡪 Backpropagation through time (13)
25
Unit delay
How do we train RNNs? 🡪 Backpropagation through time (14)
26
Unit delay
How do we train RNNs? 🡪 Backpropagation through time (15)
27
Unit delay
How do we train RNNs? 🡪 Backpropagation through time (16)
28
How do we train RNNs? 🡪 Backpropagation through time (17)
29
How do we train RNNs? 🡪 Backpropagation through time (18)
30
Gradients exploding and vanishing problem (1)
31
Gradients exploding and vanishing problem (2)
32
How to mitigate exploding/vanishing gradient problem?
33
34
The types of problems RNNs solve (1)
35
The types of problems RNNs solve (2)
36
RNNs vs. state-space models
References
(utilized for preparation of lecture notes or MATLAB code)
37