2 of 17

Sigmoid

loss

(1) o=a₀*w

(2) l=o+b

(3) a₁=sigmoid(l)

(4) E=loss(a₁)

Derivatives (chain rule)

Gate derivatives

Network update (learning rate, alpha)

Network

forward

backward prop

a₀

Given from the pre computed derivative

Given

(1)\frac{\partial{E}}{\partial{a_1}}

(2)\frac{\partial{E}}{\partial{l}}={\color{blue}\frac{\partial{E}}{\partial{a_1}}}\frac{\partial{a_1}}{\partial{l}}

(3)\frac{\partial{E}}{\partial{b}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{b}}

(4)\frac{\partial{E}}{\partial{o}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{o}}

(5)\frac{\partial{E}}{\partial{w_2}}=\frac{\partial{E}}{\partial{o_2}}\frac{\partial{o_2}}{\partial{w_2}}

=== pre-computed- derivative

l=o+b,~\frac{\partial{l}}{\partial{o}}=1,~\frac{\partial{l}}{\partial{b}}=1

o=aw,~\frac{\partial{o}}{\partial{w}}=a,~\frac{\partial{o}}{\partial{a}}=w

o=sigmoid(l)=\frac{1}{1+e^{-l}},~\frac{\partial{a}}{\partial{l}}=a(1-a)

E=\frac{1}{2}(a-t)^2,~ \frac{\partial{E}}{\partial{a}}=a-t

E=-\sum{t*log(a)+(1-t)*log(1-a)}, ~ \frac{\partial{E}}{\partial{a}}=\frac{a-t}{a(1-a)}

=== Solutions

(1)\frac{\partial{E}}{\partial{a_1}}=\frac{a_1-t}{a_1(1-a_1)}

(2)\frac{\partial{E}}{\partial{l}}=\frac{\partial{E}}{\partial{a_1}}{\color{cyan}\frac{\partial{a_1}}{\partial{l}}}=\frac{a_1-t}{a_1(1-a_1)}*{\color{cyan} a_1(1-a_1)}=a_1-t

(3)\frac{\partial{E}}{\partial{b}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{b}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}=a_1-t

(4)\frac{\partial{E}}{\partial{o}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{o}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}=a_1-t

(5)\frac{\partial{E}}{\partial{w}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}\frac{\partial{o}}{\partial{w}}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}a_0}

=== Network update

w = w - \alpha \frac{\partial{E}}{\partial{w}}

b = b - \alpha \frac{\partial{E}}{\partial{b}}

3 of 17

Too much?

Then, let’s go one by one

4 of 17

a1 = sigmoid (w*a0 + b)

Forward pass, OK? Read (1), (2), ...

Sigmoid

loss

(2) l=o+b

(3) a₁=sigmoid(l)

(4) E=loss(a₁)

Network

forward

a₀

(1) o=a₀*w

(1)\frac{\partial{E}}{\partial{a_1}}

(2)\frac{\partial{E}}{\partial{l}}={\color{blue}\frac{\partial{E}}{\partial{a_1}}}\frac{\partial{a_1}}{\partial{l}}

(3)\frac{\partial{E}}{\partial{b}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{b}}

(4)\frac{\partial{E}}{\partial{o}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{o}}

(5)\frac{\partial{E}}{\partial{w_2}}=\frac{\partial{E}}{\partial{o_2}}\frac{\partial{o_2}}{\partial{w_2}}

=== pre-computed- derivative

l=o+b,~\frac{\partial{l}}{\partial{o}}=1,~\frac{\partial{l}}{\partial{b}}=1

o=aw,~\frac{\partial{o}}{\partial{w}}=a,~\frac{\partial{o}}{\partial{a}}=w

o=sigmoid(l)=\frac{1}{1+e^{-l}},~\frac{\partial{a}}{\partial{l}}=a(1-a)

E=\frac{1}{2}(a-t)^2,~ \frac{\partial{E}}{\partial{a}}=a-t

E=-\sum{t*log(a)+(1-t)*log(1-a)}, ~ \frac{\partial{E}}{\partial{a}}=\frac{a-t}{a(1-a)}

=== Solutions

(1)\frac{\partial{E}}{\partial{a_1}}=\frac{a_1-t}{a_1(1-a_1)}

(2)\frac{\partial{E}}{\partial{l}}=\frac{\partial{E}}{\partial{a_1}}{\color{cyan}\frac{\partial{a_1}}{\partial{l}}}=\frac{a_1-t}{a_1(1-a_1)}*{\color{cyan} a_1(1-a_1)=a_1-t}

(3)\frac{\partial{E}}{\partial{b}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{b}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(4)\frac{\partial{E}}{\partial{o}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{o}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(5)\frac{\partial{E}}{\partial{w}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}\frac{\partial{o}}{\partial{w}}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}a_0}

=== Network update

w = w - \alpha \frac{\partial{E}}{\partial{w}}

b = b - \alpha \frac{\partial{E}}{\partial{b}}

5 of 17

Let’s do back propagation!

will be given. What would be

We can use the chain rule.

Sigmoid

loss

(2) l=o+b

(3) a₁=sigmoid(l)

(4) E=loss(a₁)

Network

forward

a₀

backward prop

(1) o=a₀*w

(1)\frac{\partial{E}}{\partial{a_1}}

(2)\frac{\partial{E}}{\partial{l}}={\color{blue}\frac{\partial{E}}{\partial{a_1}}}\frac{\partial{a_1}}{\partial{l}}

(3)\frac{\partial{E}}{\partial{b}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{b}}

(4)\frac{\partial{E}}{\partial{o}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{o}}

(5)\frac{\partial{E}}{\partial{w_2}}=\frac{\partial{E}}{\partial{o_2}}\frac{\partial{o_2}}{\partial{w_2}}

=== pre-computed- derivative

l=o+b,~\frac{\partial{l}}{\partial{o}}=1,~\frac{\partial{l}}{\partial{b}}=1

o=aw,~\frac{\partial{o}}{\partial{w}}=a,~\frac{\partial{o}}{\partial{a}}=w

o=sigmoid(l)=\frac{1}{1+e^{-l}},~\frac{\partial{a}}{\partial{l}}=a(1-a)

E=\frac{1}{2}(a-t)^2,~ \frac{\partial{E}}{\partial{a}}=a-t

E=-\sum{t*log(a)+(1-t)*log(1-a)}, ~ \frac{\partial{E}}{\partial{a}}=\frac{a-t}{a(1-a)}

=== Solutions

(1)\frac{\partial{E}}{\partial{a_1}}=\frac{a_1-t}{a_1(1-a_1)}

(2)\frac{\partial{E}}{\partial{l}}=\frac{\partial{E}}{\partial{a_1}}{\color{cyan}\frac{\partial{a_1}}{\partial{l}}}=\frac{a_1-t}{a_1(1-a_1)}*{\color{cyan} a_1(1-a_1)=a_1-t}

(3)\frac{\partial{E}}{\partial{b}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{b}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(4)\frac{\partial{E}}{\partial{o}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{o}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(5)\frac{\partial{E}}{\partial{w}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}\frac{\partial{o}}{\partial{w}}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}a_0}

=== Network update

w = w - \alpha \frac{\partial{E}}{\partial{w}}

b = b - \alpha \frac{\partial{E}}{\partial{b}}

6 of 17

In the same manner, we can get back prop (3), (4), and (5)!

Sigmoid

loss

(2) l=o+b

(3) a₁=sigmoid(l)

(4) E=loss(a₁)

Network

forward

backward prop

a₀

(1) o=a₀*w

(1)\frac{\partial{E}}{\partial{a_1}}

(2)\frac{\partial{E}}{\partial{l}}={\color{blue}\frac{\partial{E}}{\partial{a_1}}}\frac{\partial{a_1}}{\partial{l}}

(3)\frac{\partial{E}}{\partial{b}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{b}}

(4)\frac{\partial{E}}{\partial{o}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{o}}

(5)\frac{\partial{E}}{\partial{w_2}}=\frac{\partial{E}}{\partial{o_2}}\frac{\partial{o_2}}{\partial{w_2}}

=== pre-computed- derivative

l=o+b,~\frac{\partial{l}}{\partial{o}}=1,~\frac{\partial{l}}{\partial{b}}=1

o=aw,~\frac{\partial{o}}{\partial{w}}=a,~\frac{\partial{o}}{\partial{a}}=w

o=sigmoid(l)=\frac{1}{1+e^{-l}},~\frac{\partial{a}}{\partial{l}}=a(1-a)

E=\frac{1}{2}(a-t)^2,~ \frac{\partial{E}}{\partial{a}}=a-t

E=-\sum{t*log(a)+(1-t)*log(1-a)}, ~ \frac{\partial{E}}{\partial{a}}=\frac{a-t}{a(1-a)}

=== Solutions

(1)\frac{\partial{E}}{\partial{a_1}}=\frac{a_1-t}{a_1(1-a_1)}

(2)\frac{\partial{E}}{\partial{l}}=\frac{\partial{E}}{\partial{a_1}}{\color{cyan}\frac{\partial{a_1}}{\partial{l}}}=\frac{a_1-t}{a_1(1-a_1)}*{\color{cyan} a_1(1-a_1)=a_1-t}

(3)\frac{\partial{E}}{\partial{b}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{b}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(4)\frac{\partial{E}}{\partial{o}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{o}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(5)\frac{\partial{E}}{\partial{w}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}\frac{\partial{o}}{\partial{w}}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}a_0}

=== Network update

w = w - \alpha \frac{\partial{E}}{\partial{w}}

b = b - \alpha \frac{\partial{E}}{\partial{b}}

7 of 17

These derivatives for gates will be given.

We can just use them.

Sigmoid

loss

(2) l=o+b

(3) a₁=sigmoid(l)

(4) E=loss(a₁)

Gate derivatives

Network

forward

a₀

(1) o=a₀*w

(1)\frac{\partial{E}}{\partial{a_1}}

(2)\frac{\partial{E}}{\partial{l}}={\color{blue}\frac{\partial{E}}{\partial{a_1}}}\frac{\partial{a_1}}{\partial{l}}

(3)\frac{\partial{E}}{\partial{b}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{b}}

(4)\frac{\partial{E}}{\partial{o}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{o}}

(5)\frac{\partial{E}}{\partial{w_2}}=\frac{\partial{E}}{\partial{o_2}}\frac{\partial{o_2}}{\partial{w_2}}

=== pre-computed- derivative

l=o+b,~\frac{\partial{l}}{\partial{o}}=1,~\frac{\partial{l}}{\partial{b}}=1

o=aw,~\frac{\partial{o}}{\partial{w}}=a,~\frac{\partial{o}}{\partial{a}}=w

o=sigmoid(l)=\frac{1}{1+e^{-l}},~\frac{\partial{a}}{\partial{l}}=a(1-a)

E=\frac{1}{2}(a-t)^2,~ \frac{\partial{E}}{\partial{a}}=a-t

E=-\sum{t*log(a)+(1-t)*log(1-a)}, ~ \frac{\partial{E}}{\partial{a}}=\frac{a-t}{a(1-a)}

=== Solutions

(1)\frac{\partial{E}}{\partial{a_1}}=\frac{a_1-t}{a_1(1-a_1)}

(2)\frac{\partial{E}}{\partial{l}}=\frac{\partial{E}}{\partial{a_1}}{\color{cyan}\frac{\partial{a_1}}{\partial{l}}}=\frac{a_1-t}{a_1(1-a_1)}*{\color{cyan} a_1(1-a_1)=a_1-t}

(3)\frac{\partial{E}}{\partial{b}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{b}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(4)\frac{\partial{E}}{\partial{o}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{o}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(5)\frac{\partial{E}}{\partial{w}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}\frac{\partial{o}}{\partial{w}}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}a_0}

=== Network update

w = w - \alpha \frac{\partial{E}}{\partial{w}}

b = b - \alpha \frac{\partial{E}}{\partial{b}}

8 of 17

Just apply them one by one and solve each derivative one by one!

Sigmoid

loss

(2) l=o+b

(3) a₁=sigmoid(l)

(4) E=loss(a₁)

Derivatives (chain rule)

Gate derivatives

Network

forward

backward prop

a₀

Given from the pre computed derivative

Given

(1) o=a₀*w

(1)\frac{\partial{E}}{\partial{a_1}}

(2)\frac{\partial{E}}{\partial{l}}={\color{blue}\frac{\partial{E}}{\partial{a_1}}}\frac{\partial{a_1}}{\partial{l}}

(3)\frac{\partial{E}}{\partial{b}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{b}}

(4)\frac{\partial{E}}{\partial{o}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{o}}

(5)\frac{\partial{E}}{\partial{w_2}}=\frac{\partial{E}}{\partial{o_2}}\frac{\partial{o_2}}{\partial{w_2}}

=== pre-computed- derivative

l=o+b,~\frac{\partial{l}}{\partial{o}}=1,~\frac{\partial{l}}{\partial{b}}=1

o=aw,~\frac{\partial{o}}{\partial{w}}=a,~\frac{\partial{o}}{\partial{a}}=w

o=sigmoid(l)=\frac{1}{1+e^{-l}},~\frac{\partial{a}}{\partial{l}}=a(1-a)

E=\frac{1}{2}(a-t)^2,~ \frac{\partial{E}}{\partial{a}}=a-t

E=-\sum{t*log(a)+(1-t)*log(1-a)}, ~ \frac{\partial{E}}{\partial{a}}=\frac{a-t}{a(1-a)}

=== Solutions

(1)\frac{\partial{E}}{\partial{a_1}}=\frac{a_1-t}{a_1(1-a_1)}

(2)\frac{\partial{E}}{\partial{l}}=\frac{\partial{E}}{\partial{a_1}}{\color{cyan}\frac{\partial{a_1}}{\partial{l}}}=\frac{a_1-t}{a_1(1-a_1)}*{\color{cyan} a_1(1-a_1)=a_1-t}

(3)\frac{\partial{E}}{\partial{b}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{b}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(4)\frac{\partial{E}}{\partial{o}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{o}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(5)\frac{\partial{E}}{\partial{w}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}\frac{\partial{o}}{\partial{w}}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}a_0}

=== Network update

w = w - \alpha \frac{\partial{E}}{\partial{w}}

b = b - \alpha \frac{\partial{E}}{\partial{b}}

9 of 17

Matrix

Sigmoid

loss

(2) l=o+b

(3) a₁=sigmoid(l)

(4) E=loss(a₁)

Derivatives (chain rule)

Gate derivatives

Network update (learning rate, alpha)

Network

forward

backward prop

a₀

(1) o=a₀*w

(1)\frac{\partial{E}}{\partial{a_1}}

(2)\frac{\partial{E}}{\partial{l}}={\color{blue}\frac{\partial{E}}{\partial{a_1}}}\frac{\partial{a_1}}{\partial{l}}

(3)\frac{\partial{E}}{\partial{b}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{b}}

(4)\frac{\partial{E}}{\partial{o}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{o}}

(5)\frac{\partial{E}}{\partial{w_2}}=\frac{\partial{E}}{\partial{o_2}}\frac{\partial{o_2}}{\partial{w_2}}

=== pre-computed- derivative

l=o+b,~\frac{\partial{l}}{\partial{o}}=1,~\frac{\partial{l}}{\partial{b}}=1

o=aw,~\frac{\partial{o}}{\partial{w}}=a,~\frac{\partial{o}}{\partial{a}}=w

o=sigmoid(l)=\frac{1}{1+e^{-l}},~\frac{\partial{a}}{\partial{l}}=a(1-a)

E=\frac{1}{2}(a-t)^2,~ \frac{\partial{E}}{\partial{a}}=a-t

E=-\sum{t*log(a)+(1-t)*log(1-a)}, ~ \frac{\partial{E}}{\partial{a}}=\frac{a-t}{a(1-a)}

=== Solutions

(1)\frac{\partial{E}}{\partial{a_1}}=\frac{a_1-t}{a_1(1-a_1)}

(2)\frac{\partial{E}}{\partial{l}}=\frac{\partial{E}}{\partial{a_1}}{\color{cyan}\frac{\partial{a_1}}{\partial{l}}}=\frac{a_1-t}{a_1(1-a_1)}*{\color{cyan} a_1(1-a_1)=a_1-t}

(3)\frac{\partial{E}}{\partial{b}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{b}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(4)\frac{\partial{E}}{\partial{o}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{o}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(5)\frac{\partial{E}}{\partial{w}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}\frac{\partial{o}}{\partial{w}}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}a_0}

(5)\frac{\partial{E}}{\partial{w}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}\frac{\partial{o}}{\partial{w}}}={\color{cyan}a_0^T}\frac{\partial{E}}{\partial{o}}={\color{cyan}a_0^T}(a_1-t)

=== Network update

w = w - \alpha \frac{\partial{E}}{\partial{w}}

b = b - \alpha \frac{\partial{E}}{\partial{b}}

10 of 17

Done! Let’s update our network using derivatives!

Sigmoid

loss

(2) l=o+b

(3) a₁=sigmoid(l)

(4) E=loss(a₁)

Derivatives (chain rule)

Gate derivatives

Network update (learning rate, alpha)

Network

forward

backward prop

a₀

(1) o=a₀*w

(1)\frac{\partial{E}}{\partial{a_1}}

(2)\frac{\partial{E}}{\partial{l}}={\color{blue}\frac{\partial{E}}{\partial{a_1}}}\frac{\partial{a_1}}{\partial{l}}

(3)\frac{\partial{E}}{\partial{b}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{b}}

(4)\frac{\partial{E}}{\partial{o}}={\color{blue}\frac{\partial{E}}{\partial{l}}}\frac{\partial{l}}{\partial{o}}

(5)\frac{\partial{E}}{\partial{w_2}}=\frac{\partial{E}}{\partial{o_2}}\frac{\partial{o_2}}{\partial{w_2}}

=== pre-computed- derivative

l=o+b,~\frac{\partial{l}}{\partial{o}}=1,~\frac{\partial{l}}{\partial{b}}=1

o=aw,~\frac{\partial{o}}{\partial{w}}=a,~\frac{\partial{o}}{\partial{a}}=w

o=sigmoid(l)=\frac{1}{1+e^{-l}},~\frac{\partial{a}}{\partial{l}}=a(1-a)

E=\frac{1}{2}(a-t)^2,~ \frac{\partial{E}}{\partial{a}}=a-t

E=-\sum{t*log(a)+(1-t)*log(1-a)}, ~ \frac{\partial{E}}{\partial{a}}=\frac{a-t}{a(1-a)}

=== Solutions

(1)\frac{\partial{E}}{\partial{a_1}}=\frac{a_1-t}{a_1(1-a_1)}

(2)\frac{\partial{E}}{\partial{l}}=\frac{\partial{E}}{\partial{a_1}}{\color{cyan}\frac{\partial{a_1}}{\partial{l}}}=\frac{a_1-t}{a_1(1-a_1)}*{\color{cyan} a_1(1-a_1)=a_1-t}

(3)\frac{\partial{E}}{\partial{b}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{b}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(4)\frac{\partial{E}}{\partial{o}}=\frac{\partial{E}}{\partial{l}}{\color{cyan}\frac{\partial{l}}{\partial{o}}}

=\frac{\partial{E}}{\partial{l}}*{\color{cyan}1}

(5)\frac{\partial{E}}{\partial{w}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}\frac{\partial{o}}{\partial{w}}}=\frac{\partial{E}}{\partial{o}}{\color{cyan}a_0}

=== Network update

w = w - \alpha \frac{\partial{E}}{\partial{w}}

b = b - \alpha \frac{\partial{E}}{\partial{b}}

11 of 17

Now got it, but how about N layers?

They are the same, but just more passes!

12 of 17

Sigmoid

loss

(1) o₁=a₀w₁

a₀

w₁

b₁

w₂

b₂

(2) l₁=o₁+b₁

(3) a₁=sigmoid(l₁)

(4) o₂=a₁w₂

(5) l₂=o₂+b₂

(6) a₂=sigmoid(l₂)

(7) E

Derivatives (chain rule)

Gate derivatives

Network update (learning rate, alpha)

Network

forward

backward prop

(1)\frac{\partial{E}}{\partial{a_2}}

(2)\frac{\partial{E}}{\partial{l_2}}=\frac{\partial{E}}{\partial{a_2}}\frac{\partial{a_2}}{\partial{l_2}}

(3)\frac{\partial{E}}{\partial{b_2}}=\frac{\partial{E}}{\partial{l_2}}\frac{\partial{l_2}}{\partial{b_2}}

(4)\frac{\partial{E}}{\partial{o_2}}=\frac{\partial{E}}{\partial{l_2}}\frac{\partial{l_2}}{\partial{o_2}}

(5)\frac{\partial{E}}{\partial{w_2}}=\frac{\partial{E}}{\partial{o_2}}\frac{\partial{o_2}}{\partial{w_2}}

(6)\frac{\partial{E}}{\partial{a_1}}=\frac{\partial{E}}{\partial{o_2}}\frac{\partial{o_2}}{\partial{a_1}}

(7)\frac{\partial{E}}{\partial{l_1}}=\frac{\partial{E}}{\partial{a_1}}\frac{\partial{a_1}}{\partial{l_1}}

(8)\frac{\partial{E}}{\partial{b_1}}=\frac{\partial{E}}{\partial{l_1}}\frac{\partial{l_1}}{\partial{b_1}}

(9)\frac{\partial{E}}{\partial{o_1}}=\frac{\partial{E}}{\partial{l_1}}\frac{\partial{l_1}}{\partial{o_1}}

(10)\frac{\partial{E}}{\partial{w_1}}=\frac{\partial{E}}{\partial{o_1}}\frac{\partial{o_1}}{\partial{w_1}}

=== pre-computed- derivative

l=o+b,~\frac{\partial{l}}{\partial{o}}=1,~\frac{\partial{l}}{\partial{b}}=1

o=aw,~\frac{\partial{o}}{\partial{w}}=a,~\frac{\partial{o}}{\partial{a}}=w

a=sigmoid(l)=\frac{1}{1+e^{-l}},~\frac{\partial{a}}{\partial{l}}=a(1-a)

E=\frac{1}{2}(a-t)^2,~ \frac{\partial{E}}{\partial{a}}=a-t

E=-\sum{t*log(a)+(1-t)*log(1-a)}, ~ \frac{\partial{E}}{\partial{a}}=\frac{a-t}{a(1-a)}

=== Solutions

(1)\frac{\partial{E}}{\partial{a_2}}=\frac{a_2-t}{a_2(1-a_2)}

(2)\frac{\partial{E}}{\partial{l_2}}=\frac{\partial{E}}{\partial{a_2}}\frac{\partial{a_2}}{\partial{l_2}}=\frac{a_2-t}{a_2(1-a_2)}*a_2(1-a_2)=a_2-t

(3)\frac{\partial{E}}{\partial{b_2}}=\frac{\partial{E}}{\partial{l_2}}\frac{\partial{l_2}}{\partial{b_2}}

=\frac{\partial{E}}{\partial{l_2}}*1

(4)\frac{\partial{E}}{\partial{o_2}}=\frac{\partial{E}}{\partial{l_2}}\frac{\partial{l_2}}{\partial{o_2}}

=\frac{\partial{E}}{\partial{l_2}}*1

(5)\frac{\partial{E}}{\partial{w_2}}=\frac{\partial{E}}{\partial{o_2}}\frac{\partial{o_2}}{\partial{w_2}}=\frac{\partial{E}}{\partial{o_2}}a_1

(6)\frac{\partial{E}}{\partial{a_1}}=\frac{\partial{E}}{\partial{o_2}}\frac{\partial{o_2}}{\partial{a_1}}=\frac{\partial{E}}{\partial{o_2}}w_2

(7)\frac{\partial{E}}{\partial{l_1}}=\frac{\partial{E}}{\partial{a_1}}\frac{\partial{a_1}}{\partial{l_1}}=\frac{\partial{E}}{\partial{a_1}}a_1(1-a_1)

(8)\frac{\partial{E}}{\partial{b_1}}=\frac{\partial{E}}{\partial{l_1}}\frac{\partial{l_1}}{\partial{b_1}}=\frac{\partial{E}}{\partial{l_1}}*1(9)\frac{\partial{E}}{\partial{o_1}}=\frac{\partial{E}}{\partial{l_1}}\frac{\partial{l_1}}{\partial{o_1}}=\frac{\partial{E}}{\partial{l_1}}*1

(10)\frac{\partial{E}}{\partial{w_1}}=\frac{\partial{E}}{\partial{o_1}}\frac{\partial{o_1}}{\partial{w_1}}=\frac{\partial{E}}{\partial{o_1}}a_1

=== Network update

w_1 = w_1 - \alpha \frac{\partial{E}}{\partial{w_1}}, w_2 = w_2 - \alpha \frac{\partial{E}}{\partial{w_2}}

b_1 = b_1 - \alpha \frac{\partial{E}}{\partial{b_1}}, b_2 = b_2 - \alpha \frac{\partial{E}}{\partial{b_2}}

13 of 17

That was for single values. How about matrix?

Almost the same. Just see the next!

14 of 17

Sigmoid

loss

(1) O₁=A₀W₁

A₀

W₁

B₁

W₂

B₂

(2) L₁=O₁+B₁

(3) A₁=sigmoid(L₁)

(4) O₂=A₁W₂

(5) L₂=O₂+B₂

(6) A₂=sigmoid(L₂)

(7) E

Derivatives (chain rule)

Gate derivatives

Network update (learning rate, alpha)

Network

forward

backward prop

(1)\frac{\partial{E}}{\partial{A_2}}

(2)\frac{\partial{E}}{\partial{L_2}}=\frac{\partial{E}}{\partial{A_2}}\frac{\partial{A_2}}{\partial{L_2}}

(3)\frac{\partial{E}}{\partial{B_2}}=\frac{\partial{E}}{\partial{L_2}}\frac{\partial{L_2}}{\partial{B_2}}

(4)\frac{\partial{E}}{\partial{O_2}}=\frac{\partial{E}}{\partial{L_2}}\frac{\partial{L_2}}{\partial{O_2}}

(5)\frac{\partial{E}}{\partial{W_2}}=\frac{\partial{E}}{\partial{O_2}}\frac{\partial{O_2}}{\partial{W_2}}

(6)\frac{\partial{E}}{\partial{A_1}}=\frac{\partial{E}}{\partial{O_2}}\frac{\partial{O_2}}{\partial{A_1}}

(7)\frac{\partial{E}}{\partial{L_1}}=\frac{\partial{E}}{\partial{A_1}}\frac{\partial{A_1}}{\partial{L_1}}

(8)\frac{\partial{E}}{\partial{B_1}}=\frac{\partial{E}}{\partial{L_1}}\frac{\partial{L_1}}{\partial{B_1}}

(9)\frac{\partial{E}}{\partial{O_1}}=\frac{\partial{E}}{\partial{L_1}}\frac{\partial{L_1}}{\partial{O_1}}

(10)\frac{\partial{E}}{\partial{w_1}}=\frac{\partial{E}}{\partial{o_1}}\frac{\partial{o_1}}{\partial{w_1}}

=== pre-computed- derivative

l=o+b,~\frac{\partial{l}}{\partial{o}}=1,~\frac{\partial{l}}{\partial{b}}=1

\frac{\partial{E}}{\partial{W}}=\frac{\partial{E}}{\partial{O}}\frac{\partial{O}}{\partial{W}}=A^T\frac{\partial{E}}{\partial{O}}, where~O=AW

\frac{\partial{E}}{\partial{A}}=\frac{\partial{E}}{\partial{O}}\frac{\partial{O}}{\partial{A}}=\frac{\partial{E}}{\partial{O}}W^T, where~O=AW

a=sigmoid(l)=\frac{1}{1+e^{-l}},~\frac{\partial{a}}{\partial{l}}=a(1-a)

E=\frac{1}{2}(a-t)^2,~ \frac{\partial{E}}{\partial{a}}=a-t

E=-\sum{t*log(a)+(1-t)*log(1-a)}, ~ \frac{\partial{E}}{\partial{a}}=\frac{a-t}{a(1-a)}

=== Solutions

(1)\frac{\partial{E}}{\partial{A_2}}=\frac{a_2-t}{a_2(1-a_2)}

(2)\frac{\partial{E}}{\partial{L_2}}=\frac{\partial{E}}{\partial{A_2}}\frac{\partial{A_2}}{\partial{L_2}}=\frac{A_2-T}{A_2(1-A_2)}*A_2(1-A_2)=A_2-T

(3)\frac{\partial{E}}{\partial{B_2}}=\frac{\partial{E}}{\partial{L_2}}\frac{\partial{L_2}}{\partial{B_2}}

=\frac{\partial{E}}{\partial{L_2}}*1

(4)\frac{\partial{E}}{\partial{o_2}}=\frac{\partial{E}}{\partial{l_2}}\frac{\partial{l_2}}{\partial{o_2}}

=\frac{\partial{E}}{\partial{l_2}}*1

(5)\frac{\partial{E}}{\partial{W_2}}=\frac{\partial{E}}{\partial{O_2}}\frac{\partial{O_2}}{\partial{W_2}}=A_1^T\frac{\partial{E}}{\partial{O_2}}

(6)\frac{\partial{E}}{\partial{A_1}}=\frac{\partial{E}}{\partial{O_2}}\frac{\partial{O_2}}{\partial{A_1}}=\frac{\partial{E}}{\partial{O_2}}W_2^T

(7)\frac{\partial{E}}{\partial{l_1}}=\frac{\partial{E}}{\partial{a_1}}\frac{\partial{a_1}}{\partial{l_1}}=\frac{\partial{E}}{\partial{a_1}}a_1(1-a_1)

(8)\frac{\partial{E}}{\partial{B_1}}=\frac{\partial{E}}{\partial{L_1}}\frac{\partial{L_1}}{\partial{B_1}}=\frac{\partial{E}}{\partial{L_1}}*1

(9)\frac{\partial{E}}{\partial{o_1}}=\frac{\partial{E}}{\partial{l_1}}\frac{\partial{l_1}}{\partial{o_1}}=\frac{\partial{E}}{\partial{l_1}}*1

(10)\frac{\partial{E}}{\partial{W_1}}=\frac{\partial{E}}{\partial{O_1}}\frac{\partial{O_1}}{\partial{W_1}}={\color{blue}A_0^T}\frac{\partial{E}}{\partial{O_1}}

=== Network update

W_1 = W_1 - \alpha \frac{\partial{E}}{\partial{W_1}}, W_2 = W_2 - \alpha \frac{\partial{E}}{\partial{W_2}}

B_1 = B_1 - \alpha \frac{\partial{E}}{\partial{B_1}}, B_2 = B_2 - \alpha \frac{\partial{E}}{\partial{B_2}}

1 of 17

2 of 17

3 of 17

4 of 17

5 of 17

6 of 17

7 of 17

8 of 17

9 of 17

10 of 17

11 of 17

12 of 17

13 of 17

14 of 17

15 of 17

16 of 17

17 of 17