1 of 111

Neural Network Architectures for Time-Series

2 of 111

So Far

Regression, Classification, Dimension Reduction,
Based on snapshot-type data

2

3 of 111

Robocup 2011

3

Robocup 2011 Final

https://youtu.be/VPOAyElEegA

4 of 111

Sequence Matters

4

5 of 111

What is a Sequence ?

Sentence

“This morning I took the dog for a walk.”

Medical signals

5

Speech waveform

Vibration measurement

6 of 111

Sequence Modeling

Most of the real-world data is time-series

There are important bits to be considered

Past events
Relationship between events

Causality
Credit assignment

Learning the structure and hierarchy

Use the past and present observations to predict the future

6

7 of 111

(Deterministic) Time Series Data

For example

Closed-form

Linear difference equation (LDE) and initial condition

High order LDEs

7

8 of 111

(Stochastic) Time Series Data

Stationary

Non-stationary

Mean and variance change over time

8

9 of 111

Dealing with Non-Stationarity

Linear trends

9

10 of 111

Dealing with Non-Stationarity

Non-linear trends

10

11 of 111

Dealing with Non-Stationarity

Seasonal trends

11

12 of 111

Dealing with Non-Stationarity

Model assumption

12

13 of 111

Why Time-Series Data

(almost) all the data coming from manufacturing environment are time-series data

sensor data,
process times,
material measurement,
equipment maintenance history,
image data, etc.

Manufacturing application is about one of the following:

prediction of time-series values
anomaly detection on time-series data
classification of time-series values
metrology and inspection

13

14 of 111

Time-series Data

14

15 of 111

Supervised and Unsupervised Learning for Time-series

For supervised learning, we define two time series

Supervised time-series learning

Unsupervised time-series anomaly detection

Find time segment that is considerably different from the rest

15

16 of 111

Markov Process

16

17 of 111

Sequential Processes

Most classifiers ignored the sequential aspects of data

Consider a system which can occupy one of N discrete states or categories

We are interested in stochastic systems, in which state evolution is random
Any joint distribution can be factored into a series of conditional distributions

17

Almost impossible to compute !

18 of 111

Markov Chain

18

19 of 111

State Transition Matrix

19

20 of 111

State Transition Matrix

20

s₁

s₃

s₂

1/2

1/3

2/3

1

21 of 111

Example: MC Episodes

21

s₁

s₃

s₂

1/2

1/3

2/3

1

22 of 111

Hidden Markov Model

22

23 of 111

Hidden Markov Models

Discrete state-space model

Used in speech recognition
State representation is simple
Hard to scale-up the training

Assumption

We can observe something that’s affected by the true state
Natural way of thinking

Limited sensors (incomplete state information)

But still partially related

Noisy sensors

Unreliable

23

24 of 111

Hidden Markov Model (HMM)

24

25 of 111

Kalman Filter

25

26 of 111

Kalman Filter

Linear dynamical system of motion

A, B, C ?

Continuous State space model

For filtering and control applications
Linear-Gaussian state space model
Widely used in many applications:

GPS, weather systems, etc.

Weakness

Linear state space model assumed
Difficult to apply to highly non-linear domains

26

27 of 111

Kalman Filter

27

https://youtu.be/aIivvB4UvdQ

28 of 111

Neural Network Architectures for Time-Series:�Recurrent Neural Network (RNN)

Most slides from CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

29 of 111

Recurrent NN (RNN)

Hidden state extraction and transformation

29

Y_n-1

Y_n

Y_n+1

O_n+1

Classification

30 of 111

Recurrent NN (RNN)

Hidden state extraction and transformation

30

Y_n-1

Y_n

Y_n+1

X_n+1

X_n

X_n-1

O_n+1

Learned latent state

Classification based on states

U

31 of 111

Recurrent NN (RNN)

Hidden state extraction and transformation
Good for sequential data (dynamic behavior)

31

Y_n-1

Y_n

Y_n+1

X_n+1

X_n

X_n-1

O_n+1

…

Learned latent state and its dynamics

Classification based on states

W

U

W

U

32 of 111

Recurrent NN

32

33 of 111

Representation Shortcut

Input at each time is a vector
Each layer has many neurons

Output layer too may have many neurons

But will represent everything simple boxes

Each box actually represents an entire layer with many units

33

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

34 of 111

Representation Shortcut

Input at each time is a vector
Each layer has many neurons

Output layer too may have many neurons

But will represent everything simple boxes

Each box actually represents an entire layer with many units

34

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

35 of 111

Representation Shortcut

Input at each time is a vector
Each layer has many neurons

Output layer too may have many neurons

But will represent everything simple boxes

Each box actually represents an entire layer with many units

35

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

36 of 111

Sliding Predictor

The sliding predictor

Look at the last few days
This is just a convolutional neural net applied to sequential data

36

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

37 of 111

Sliding Predictor

The sliding predictor

Look at the last few days
This is just a convolutional neural net applied to sequential data

37

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

38 of 111

Sliding Predictor

The sliding predictor

Look at the last few days
This is just a convolutional neural net applied to sequential data

38

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

39 of 111

Sliding Predictor

The sliding predictor

Look at the last few days
This is just a convolutional neural net applied to sequential data

39

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

40 of 111

Finite-Response Model

This is a finite response system

Something that happens today only affects the output of the system for 𝑁 days into the future
𝑁 is the width of the system

40

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

41 of 111

Finite-Response Model

Problem: Increasing the “history” makes the network more complex

41

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

42 of 111

In Theory, We Want Infinite Memory

Required: Infinite response systems

What happens today can continue to affect the output forever
Possibly with weaker and weaker influence

42

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

43 of 111

Infinite Response Systems

43

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

44 of 111

Autoregression

An autoregressive net with recursion from the output

44

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

45 of 111

Autoregression

An autoregressive net with recursion from the output

45

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

46 of 111

Autoregression

An autoregressive net with recursion from the output

46

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

47 of 111

Autoregression

An autoregressive net with recursion from the output

47

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

48 of 111

Autoregression

An autoregressive net with recursion from the output

48

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

49 of 111

Autoregression

An autoregressive net with recursion from the output

49

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

50 of 111

Autoregression

An autoregressive net with recursion from the output

50

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

51 of 111

Autoregression

An autoregressive net with recursion from the output

51

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

52 of 111

More Complete Representation

52

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

53 of 111

An Alternate Model for Infinite Response Systems

53

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

54 of 111

Single Hidden Layer RNN (Simplest State-Space Model)

54

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

55 of 111

Multiple Recurrent Layer RNN

55

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

56 of 111

The Folded Version of RNN

56

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

57 of 111

The Folded Version of RNN

57

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

58 of 111

Recurrent Neural Network

Simplified models often drawn
The loops imply recurrence

58

CMU 11-785 Intro. to Deep Learning by Prof. Bhiksha Raj

59 of 111

RNN Applications

Machine translation
Speech recognition
Text-to-speech
Image captioning
Video analysis/understanding

59

60 of 111

“Vanilla” Neural Network

60

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

61 of 111

Recurrent Neural Network: Process Sequences

61

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

62 of 111

Recurrent Neural Network: Process Sequences

62

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

63 of 111

Recurrent Neural Network: Process Sequences

63

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

64 of 111

Recurrent Neural Network: Process Sequences

64

e.g. Video classification on frame level

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

65 of 111

Recurrent Neural Networks

65

66 of 111

Unrolling the Recurrence

66

67 of 111

Unrolling the Recurrence

67

68 of 111

Unrolling the Recurrence

68

69 of 111

Unrolling the Recurrence

69

70 of 111

Unrolling the Recurrence

70

71 of 111

Unrolling the Recurrence

71

72 of 111

Unrolling the Recurrence

72

73 of 111

Unrolling the Recurrence

73

74 of 111

Unrolling the Recurrence

74

75 of 111

Unrolling the Recurrence

75

76 of 111

Recurrent Connections

76

77 of 111

Recurrent Connections

77

78 of 111

Feedforward Propagation

This is a RNN where the input and output sequences are of the same length
Feedforward operation proceeds from left to right
Update Equations:

78

79 of 111

How to Train RNN

79

80 of 111

Backward Propagation

Loss would just be the sum of losses over time steps
Treat the recurrent network as a usual multilayer network and apply backpropagation on the unrolled network
This is called Backpropagation through time (BPTT)

80

81 of 111

Long Short-Term Memory (LSTM)

81

82 of 111

Long Short-Term Memory (LSTM)

Long-Term Dependencies

Gradients propagated over many stages tend to either vanish or explode
Difficulty with long-term dependencies arises from the exponentially smaller weights given to long-term interactions
Introduce a memory state that runs through only linear operators
Use gating units to control the updates of the state

82

83 of 111

Example

“I grew up in France… I speak fluent French.”

83

https://colah.github.io/posts/2015-08-Understanding-LSTMs/

84 of 111

Long Short-Term Memory (LSTM)

Consists of a memory cell and a set of gating units

Memory cell is the context that carries over
Forget gate controls erase operation
Input gate controls write operation
Output gate controls the read operation

84

https://colah.github.io/posts/2015-08-Understanding-LSTMs/

85 of 111

Long Short-Term Memory (LSTM)

Consists of a memory cell and a set of gating units

Memory cell is the context that carries over
Forget gate controls erase operation
Input gate controls write operation
Output gate controls the read operation

85

https://colah.github.io/posts/2015-08-Understanding-LSTMs/

86 of 111

Element-by-Element

86

3
4
5

6
7
8

3+6
4+7
5+8

9
11
13

3
4
5

6
7
8

3x6
4x7
5x8

18
28
40

Element-by-Element Addition

Element-by-Element Multiplication

87 of 111

Gating

87

3
4
5

1.0
0.5
0.0

3x1.0
4x0.5
5x0.0

3.0
2.0
0.0

signal

On/Off

gating

88 of 111

Long Short-Term Memory (LSTM)

88

https://elham-khanche.github.io/blog/RNNs_and_LSTM/

89 of 111

Long Short-Term Memory (LSTM)

89

https://elham-khanche.github.io/blog/RNNs_and_LSTM/

90 of 111

Long Short-Term Memory (LSTM)

90

https://elham-khanche.github.io/blog/RNNs_and_LSTM/

91 of 111

Long Short-Term Memory (LSTM)

91

https://elham-khanche.github.io/blog/RNNs_and_LSTM/

92 of 111

Long Short-Term Memory (LSTM)

92

https://elham-khanche.github.io/blog/RNNs_and_LSTM/

93 of 111

Long Short-Term Memory (LSTM)

93

https://elham-khanche.github.io/blog/RNNs_and_LSTM/

94 of 111

Long Short-Term Memory (LSTM)

94

https://elham-khanche.github.io/blog/RNNs_and_LSTM/

95 of 111

Long Short-Term Memory

Forget gate controls erase operation
Input gate controls write operation
Output gate controls the read operation

95

96 of 111

Long Short-Term Memory

Forget gate controls erase operation
Input gate controls write operation
Output gate controls the read operation

96

97 of 111

Long Short-Term Memory

Forget gate controls erase operation
Input gate controls write operation
Output gate controls the read operation

97

98 of 111

Long Short-Term Memory

Forget gate controls erase operation
Input gate controls write operation
Output gate controls the read operation

98

99 of 111

Long Short-Term Memory

Forget gate controls erase operation
Input gate controls write operation
Output gate controls the read operation

99

100 of 111

Long Short-Term Memory

100

101 of 111

Weakness of RNN and LSTM

Sequential computation is slow

Vanishing and exploding gradients are still problematic

Long-term credit assignment is difficult

101

102 of 111

LSTM Implementation

102

103 of 111

Time Series Data and LSTM

103

104 of 111

LSTM for Classification

104

105 of 111

LSTM for Prediction

105

106 of 111

LSTM with TensorFlow

An example for predicting a next piece of an acceleration signal
Regression problem

106

107 of 111

LSTM Structure

107

108 of 111

Build a Model

Define the LSTM cells

108

109 of 111

Cost, Initializer and Optimizer

Loss

Regression: Squared loss

Optimizer

AdamOptimizer: the most popular optimize

109

110 of 111

Prediction Example

110

111 of 111

Prediction Example

111