ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
What is this spreadsheet about?
2
This spreadsheet explores different flavors/variations of Stochastic Gradient Descent, the key optimization method used in Deep Learning.
3
To keep things simple, instead of working with neural networks here, we will use a simple linear model (y=ax+b). However, the optimization methods are the same for NNs.
4
Make a copy of this spreadsheet in order to edit and save it. Note that macros only work on 'basic SGD' sheet for now.
5
6
Goal: We want to create a model so that we can predict y when we're given x.
7
To build the model, we are given training data x, y and we want to find an optimal a, b to create a model of the form y = ax + b.
8
9
The key idea of SGD:
10
1. Randomly choose some weights to start.
11
2. Use those weights to calculate a prediciton, calculate the error/loss, and then update those weights.
12
3. Repeat step 2 lots of times. Eventually we end up with some decent weights.
13
14
How does this relate to NNs? To make an analogy with CNNs: x would be our images, y would be the dog/cat labels, and a and b are the weights.
15
We are trying to find the optimal weights (a, b). We use the same optimization methods to find weights for our NNs.
16
17
Overview of the sheets in this notebook:
18
data:Here we randomly generate some training data x and y that we will use on the other sheets. In the real world, you would get your x and y data from another source. In our CNN, x is the image data and y is the category (dog or cat).
19
We create our data by arbitrarily choosing a and b (a=2, b=30), randomly generating x, and then calculating y = ax + b. We are going to pretend we don't know the true values of a and b, and see if our optimization algorithms help us find them.
20
We can get a sense of how good our answers are, since really we know what the "true" a and b are.
21
basic SGD:This is the most basic, vanilla form of stochastic gradient descent. If we were doing gradient descent (not stochastic!), we would evaluate our error on all our data, and then update the weights a, b.
22
What makes it stochastic is that we're just evaluating the error on a subset of our data (in this case, a single data point. For NNs, this is called our "mini-batch") and then updating the weights.
23
momentum:Accelerate SGD in the relevant direction and dampen side-to-side oscillations
24
It does this by combining a decaying average of previous gradients with the current gradient
25
rmsprop:Adapts the learning rate by dividing by exponentially decaying average of squared gradients
26
adam:rmsprop + momentum
27
adam_ann: adam with learning rate annealling
28
29
On each sheet, click 'reset' to return the a and b guesses to 1.
30
Click 'run' to copy the last row's updated a and b guesses to the top row, and recalculate, a few times. (Each is effectively one epoch)
31
32
Some comments have been added to the other sheets in this notebook.
33
Comments are indicated by a small red triangle in the corner of a cell. Hover your mouse over that cell to read the comment.
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100