1 of 18

ME5990

�

Forward Propagation by Example

2 of 18

Outline

  • Forward Propagation
    • Forward propagation
    • Loss Function

3 of 18

Forward Propagation

  • We have 4 dimension dataset and label
  • Let’s forward propagate a “trained” model (in the next page) and calculate the mean square error for this model

y

575

476

498

…

x1

2

3

5

​

x2

3

4

1

​

x3

1

1

4

​

x4

4

2

3

​

4 of 18

Forward Propagation Demonstration

  •  

 

 

 

 

 

 

Fully Connected

ReLU

FC

ReLU

FC

input

Hidden 1

Hidden 1a

Hidden 2

Hidden 2a

output

5 of 18

Forward Propagation

  • What is the dimension of weight matrix and bias vector for each fully connected layer?

​

6 of 18

Forward Propagation

  •  

7 of 18

Forward Propagation

  •  

8 of 18

Forward propagation

  • After ReLU

​

​

 

 

 

 

 

 

Fully Connected

ReLU

FC

ReLU

FC

input

Hidden 1

Hidden 1a

Hidden 2

Hidden 2a

output

9 of 18

Forward Propagation

  •  

10 of 18

Forward Propagation

  •  

11 of 18

Forward propagation

  • After ReLU

​

​

 

 

 

 

 

 

Fully Connected

ReLU

FC

ReLU

FC

input

Hidden 1

Hidden 1a

Hidden 2

Hidden 2a

output

12 of 18

Forward Propagation

  •  

13 of 18

Forward Propagation

  •  

14 of 18

Forward propagation

  • Forward Propagation

​

​

 

 

 

 

 

 

Fully Connected

ReLU

FC

ReLU

FC

input

Hidden 1

Hidden 1a

Hidden 2

Hidden 2a

output

15 of 18

Mean squared error (MSE)

  •  

x

​

Label y

1

2

3

2

4

5

3

6

6

16 of 18

Mean squared error (MSE)

  •  

 

 

 

 

 

 

Fully Connected

ReLU

FC

ReLU

FC

input

Hidden 1

Hidden 1a

Hidden 2

Hidden 2a

output

17 of 18

Mean square error

  •  

y

575

476

498

…

x1

2

3

5

​

x2

3

4

1

​

x3

1

1

4

​

x4

4

2

3

​

18 of 18

Summary

  • Forward Propagation is the simple procedure to get the output from the input (from left to right), with a given set of the parameter.
  • The parameter of the model will not be updated in the forward propagation
  • Activation function is needed after each of the linear layer.