Backpropagation (cont.)�
Prof. André E. Lazzaretti
Gradients
Matrix calculus
Scalar function f[] of a vector a
Matrix calculus
Scalar function f[] of a matrix A
Matrix calculus
Vector function f[] of vector a
Comparing vector and matrix
Scalar derivatives:
Matrix derivatives:
Comparing vector and matrix
Scalar derivatives:
Matrix derivatives:
Gradients
The forward pass
1. Write this as a series of
intermediate calculations
2. Compute these intermediate quantities
Gradients
The backward pass
1. Write this as a series of
intermediate calculations
2. Compute these intermediate quantities
3. Take derivatives of output with respect to intermediate quantities
The backward pass
1. Write this as a series of
intermediate calculations
2. Compute these intermediate quantities
3. Take derivatives of output with respect to intermediate quantities
The backward pass
1. Write this as a series of
intermediate calculations
2. Compute these intermediate quantities
3. Take derivatives of output with respect to intermediate quantities
The backward pass
1. Write this as a series of
intermediate calculations
2. Compute these intermediate quantities
3. Take derivatives of output with respect to intermediate quantities
Derivative of ReLU
“Indicator function”
Derivative of RELU
1. Consider:
where:
2. We could equivalently write:
3. Taking the derivative
4. We can equivalently pointwise multiply by diagonal
The backward pass
1. Write this as a series of
intermediate calculations
2. Compute these intermediate quantities
3. Take derivatives of output with respect to intermediate quantities
The backward pass
1. Write this as a series of
intermediate calculations
2. Compute these intermediate quantities
3. Take derivatives of output with respect to intermediate quantities
4. Take derivatives w.r.t.
parameters
The backward pass
1. Write this as a series of
intermediate calculations
2. Compute these intermediate quantities
3. Take derivatives of output with respect to intermediate quantities
4. Take derivatives w.r.t.
parameters
Backprop summary
Backprop summary