2 of 50

Loss function

Training dataset of I pairs of input/output examples:

Loss function or cost function measures how bad model is:

or for short:

Returns a scalar that is smaller when model maps inputs to outputs better

3 of 50

Training

Loss function:

Find the parameters that minimize the loss:

Returns a scalar that is smaller when model maps inputs to outputs better

4 of 50

Example: 1D Linear regression loss function

Loss function:

“Least squares loss function”

5 of 50

Example: 1D Linear regression training

6 of 50

Example: 1D Linear regression training

7 of 50

Example: 1D Linear regression training

8 of 50

Example: 1D Linear regression training

9 of 50

Example: 1D Linear regression training

This technique is known as gradient descent

10 of 50

Fitting models

Gradient descent algorithm
Linear regression example
Gabor model example
Stochastic gradient descent
Momentum
Adam

11 of 50

Gradient descent algorithm

12 of 50

Fitting models

Gradient descent algorithm
Linear regression example
Gabor model example
Stochastic gradient descent
Momentum
Adam

13 of 50

Gradient descent

Step 1: Compute derivatives (slopes of function) with

Respect to the parameters

14 of 50

Gradient descent

Step 1: Compute derivatives (slopes of function) with

Respect to the parameters

15 of 50

Gradient descent

Step 1: Compute derivatives (slopes of function) with

Respect to the parameters

16 of 50

Gradient descent

Step 1: Compute derivatives (slopes of function) with

Respect to the parameters

17 of 50

Gradient descent

Step 1: Compute derivatives (slopes of function) with

Respect to the parameters

Step 2: Update parameters according to rule

18 of 50

Gradient descent

Step 1: Compute derivatives (slopes of function) with

Respect to the parameters

Step 2: Update parameters according to rule

19 of 50

Gradient descent

20 of 50

Gradient descent

Step 1: Compute derivatives (slopes of function) with

Respect to the parameters

Step 2: Update parameters according to rule

21 of 50

Gradient descent

Step 1: Compute derivatives (slopes of function) with

Respect to the parameters

Step 2: Update parameters according to rule

22 of 50

Gradient descent

23 of 50

Gradient descent

24 of 50

Gradient descent

25 of 50

Convex problems

Non convex

Non-Convex

Convex

26 of 50

Convex problems

Non convex

Non-Convex

Convex

Test for convexity is that 2^nd derivative is positive everywhere

27 of 50

Convexity in higher dimensions

Test for convexity is that determinant of Hessian (2^nd derivative matrix) is positive everywhere.

28 of 50

Fitting models

Gradient descent algorithm
Linear regression example
Gabor model example
Stochastic gradient descent
Momentum
Adam

29 of 50

Gabor model

https://udlbook.github.io/udlfigures/

30 of 50

Gabor model

31 of 50

https://udlbook.github.io/udlfigures/

32 of 50

Gradient descent gets to the global minimum if we start in the right “valley”
Otherwise, descent to a local minimum
Or get stuck near a saddle point

33 of 50

Fitting models

Gradient descent algorithm
Linear regression example
Gabor model example
Stochastic gradient descent
Momentum
Adam

34 of 50

IDEA: add noise

Stochastic gradient descent
Compute gradient based on only a subset of points – a mini-batch
Work through dataset sampling without replacement
One pass though the data is called an epoch

37 of 50

Alternative view of SGD for the Gabor model with a batch size of three. a) Loss function for the entire training dataset. At each iteration, there is a probability distribution of possible parameter changes (inset shows samples). These correspond to different choices of the three batch elements. b) Loss function for one possible batch. The SGD algorithm moves in the downhill direction on this function for a distance that is determined by the learning rate and the local gradient magnitude. The current model (dashed function in inset) changes to better fit the batch data (solid function). c) A different batch creates a different loss function and results in a different update. d) For this batch, the algorithm moves downhill with respect to the batch loss function but uphill with respect to the global loss function in panel (a). This is how SGD can escape local minima.

38 of 50

Properties of SGD

Can escape from local minima
Adds noise, but still sensible updates as based on part of data
Uses all data equally
Less computationally expensive
Seems to find better solutions

In practice, SGD is used with:

Learning rate schedule – decrease learning rate over time

39 of 50

Fitting models

Gradient descent algorithm
Linear regression example
Gabor model example
Stochastic gradient descent
Momentum
Adam

40 of 50

Momentum

Exponential Moving Average:

41 of 50

Momentum

Weighted sum of this gradient and previous gradient

43 of 50

Fitting models

Gradient descent algorithm
Linear regression example
Gabor model example
Stochastic gradient descent
Momentum
Adam

44 of 50

Adaptive moment estimation (Adam)

45 of 50

Normalized gradients

Measure mean and pointwise squared gradient

Normalize:

46 of 50

Normalized gradients

Measure mean and pointwise squared gradient

Normalize:

47 of 50

Normalized gradients

Measure mean and pointwise squared gradient

Normalize:

48 of 50

Adaptive moment estimation (Adam)

Compute mean and pointwise squared gradients with momentum

Moderate near start of the sequence

Update the parameters

1 of 50

2 of 50

3 of 50

4 of 50

5 of 50

6 of 50

7 of 50

8 of 50

9 of 50

10 of 50

11 of 50

12 of 50

13 of 50

14 of 50

15 of 50

16 of 50

17 of 50

18 of 50

19 of 50

20 of 50

21 of 50

22 of 50

23 of 50

24 of 50

25 of 50

26 of 50

27 of 50

28 of 50

29 of 50

30 of 50

31 of 50

32 of 50

33 of 50

34 of 50

35 of 50

36 of 50

37 of 50

38 of 50

39 of 50

40 of 50

41 of 50

42 of 50

43 of 50

44 of 50

45 of 50

46 of 50

47 of 50

48 of 50

49 of 50

50 of 50