1 of 37

1

Institute for Data Science & Artificial Intelligence

​

Lecture A-2

Course: DSAI 525-Time Series and Forecasting with Machine Learning

Instructor: Ercan Atam

RNNs

2 of 37

List of contents for this lecture

2

  • ANNs for sequential data

​

  • RNNs

​

  • The types of problems RNNs solve

​

  • RNNs vs. state-space models

​

3 of 37

Relevant readings/watching for this lecture

3

  • Chapter 14 of W. Kuang, “Deep Learning: Principles and Implementations”, 1st ed. Hoboken, NJ, USA: John Wiley & Sons, 2026.

​

​

​

​

4 of 37

From feed-forward ANNs to RNNs

Recurrent Neural Networks (RNNs)

​

  • Traditional neural networks process inputs independently, without retaining information from previous inputs.

​

  • Sequential data (e.g., text, time series, videos) requires information from previous time steps to interpret current inputs.

​

  • Recurrent Neural Networks (RNNs) address this limitation by introducing recurrent

connections (loops).

​

  • These connections allow information from previous time steps to persist and influence future computations.

​

  • RNNs maintain a hidden state (memory) that is updated as new inputs arrive.

4

Unit delay

RNNs

5 of 37

Examples of applications for sequence models

5

6 of 37

The goal of RNNs

6

7 of 37

Used convention & notation for RNNs (1)

7

Unit delay

RNNs

8 of 37

Used convention & notation for RNNs (2)

8

Unit delay

RNNs

9 of 37

Used convention & notation for RNNs (3)

9

Unit delay

RNNs

10 of 37

Unfolded RNN block diagram

10

Q: Can we show “mathematically” the equivalence?

(b) Unfolded RNNs

…

Unit delay

(a) RNNs

11 of 37

Mathematically equivalence of the two diagrams (1)

11

(1)

Unit delay

(a) RNNs

12 of 37

Mathematically equivalence of the two diagrams (2)

12

Thus, (1) and (2) are the same for any given time point t.

(b) Unfolded RNNs

…

(2)

13 of 37

How do we train RNNs? 🡪 Backpropagation through time (1)

13

…

14 of 37

How do we train RNNs? 🡪 Backpropagation through time (2)

14

Consider the i-th sample and for simplicity of notation, omit the superscript i in the figure.

…

15 of 37

How do we train RNNs? 🡪 Backpropagation through time (3)

15

16 of 37

How do we train RNNs? 🡪 Backpropagation through time (4)

16

Unit delay

17 of 37

How do we train RNNs? 🡪 Backpropagation through time (5)

17

Unit delay

18 of 37

How do we train RNNs? 🡪 Backpropagation through time (6)

18

19 of 37

How do we train RNNs? 🡪 Backpropagation through time (7)

19

…

20 of 37

How do we train RNNs? 🡪 Backpropagation through time (8)

20

21 of 37

How do we train RNNs? 🡪 Backpropagation through time (9)

21

22 of 37

How do we train RNNs? 🡪 Backpropagation through time (10)

22

Unit delay

23 of 37

How do we train RNNs? 🡪 Backpropagation through time (11)

23

Unit delay

(14)

(15)

(16)

24 of 37

How do we train RNNs? 🡪 Backpropagation through time (12)

24

25 of 37

How do we train RNNs? 🡪 Backpropagation through time (13)

25

Unit delay

26 of 37

How do we train RNNs? 🡪 Backpropagation through time (14)

26

Unit delay

27 of 37

How do we train RNNs? 🡪 Backpropagation through time (15)

27

Unit delay

28 of 37

How do we train RNNs? 🡪 Backpropagation through time (16)

28

29 of 37

How do we train RNNs? 🡪 Backpropagation through time (17)

29

30 of 37

How do we train RNNs? 🡪 Backpropagation through time (18)

30

31 of 37

Gradients exploding and vanishing problem (1)

31

32 of 37

Gradients exploding and vanishing problem (2)

32

33 of 37

How to mitigate exploding/vanishing gradient problem?

33

34 of 37

34

The types of problems RNNs solve (1)

35 of 37

35

The types of problems RNNs solve (2)

36 of 37

36

RNNs vs. state-space models

37 of 37

References

(utilized for preparation of lecture notes or MATLAB code)

37

  • Chapter 14 of W. Kuang, “Deep Learning: Principles and Implementations”, 1st ed. Hoboken, NJ, USA: John Wiley & Sons, 2026.

​