1 of 56

Backpropagation pt. 1�

Prof. André E. Lazzaretti

lazzaretti@utfpr.edu.br

2 of 56

Loss function

Training dataset of I pairs of input/output examples:

Loss function or cost function measures how bad model is:

or for short:

Returns a scalar that is smaller when model maps inputs to outputs better

3 of 56

Example

4 of 56

Problem 1: Computing gradients

Loss: sum of individual terms:

SGD Algorithm:

Parameters:

Need to compute gradients

5 of 56

Why is this such a big deal?

A neural network is just an equation:

But it’s a huge equation, and we need to compute the derivative

for every parameter
for every point in the batch
for every iteration of gradient descent

6 of 56

Gradients

Backpropagation intuition
Toy model
Background mathematics
Backpropagation forward pass
Backpropagation backward pass
Algorithmic differentiation

7 of 56

Problem 1: Computing gradients

Loss: sum of individual terms:

SGD Algorithm:

Parameters:

Need to compute gradients

8 of 56

Algorithm to compute gradient efficiently

“Backpropagation algorithm”
Rumelhart, Hinton, and Williams (1986)

9 of 56

BackProp intuition #1: the forward pass

We want to know how a change in orange weight affects loss
We need to know the activations at each layer.

10 of 56

BackProp intuition #2: the backward pass

To calculate how a small change in a weight or bias feeding into hidden layer h₃ modifies the loss, we need to know:�

how a change in layer h₃ changes the model output f
how a change in model output changes the loss l

11 of 56

BackProp intuition #2: the backward pass

To calculate how a small change in a weight or bias feeding into hidden layer h₂ modifies the loss, we need to know:

how a change in layer h₂ affects h₃
how h₃ changes the model output
how this output changes the loss

12 of 56

BackProp intuition #2: the backward pass

To calculate how a small change in a weight or bias feeding into hidden layer h₁ modifies the loss, we need to know:

how a change in layer h₁ affects layer h₂
how a change in layer h₂ affects layer h₃
how layer h₃ changes the model output
how the model output changes the loss

13 of 56

Gradients

Backpropagation intuition
Toy model
Background mathematics
Backpropagation forward pass
Backpropagation backward pass
Algorithmic differentiation

14 of 56

Toy function

Consists of a series of functions that are composed with each other
Unlike in neural networks, it just uses scalars (not vectors)
“Activation functions” sin, exp, cos

15 of 56

Toy function

Derivatives

16 of 56

Gradients of toy function

We want to calculate:

17 of 56

Gradients of composed functions

Calculating expressions by hand:

some expressions very complicated.
obvious redundancy (look at sin terms in bottom equation)

18 of 56

Forward pass

1. Write this as a series of

intermediate calculations

2. Compute these intermediate quantities

19 of 56

Backward pass