Neural Networks from Scratch

An Algorithmic Approach

By: Ankit Gupta

(Founder/President @ DiscoverAI)

Neural Networks from Scratch

An Algorithmic Approach

By: Ankit Gupta

(Founder/President @ DiscoverAI)

Components

- Input Layer
- Hidden Layer
- Output Layer
- Weights and Biases between Layers
- Activation Function

Architecture

Neural Network Training

- Output is designated by the following function:
- Y = 𝜎(W
_{2}𝝈(W_{1}x+b_{1})+b_{2}) - Weights are represented by W
_{1}and W_{2} - Biases represented by b
_{1}and b_{2} - Two Steps in Training:
- Feedforward
- Backpropagation

Sequential Graph

Feedforward Implementation

def feedforward(self):

self.layer1 = sigmoid(np.dot(self.input, self.weights1))

self.output = sigmoid(np.dot(self.layer1, self.weights2))

Loss Function

- Many Different Possibilities
- We will choose: Sum of Squares Error =
- Goal: Minimize the Loss Function

_{}

Backpropagation

- Propagate the error back and update our weights and biases
- Finds the extent to which we need to adjust the weights and biases

def backprop(self):

d_weights2 = np.dot(self.layer1.T, (2*(self.y - self.output) * sigmoid_derivative(self.output)))

d_weights1 = np.dot(self.input.T, (np.dot(2*(self.y - self.output) * sigmoid_derivative(self.output), self.weights2.T) * sigmoid_derivative(self.layer1)))

self.weights1 += d_weights1

self.weights2 += d_weights2

Gradient Descent

- Derivative of the Loss Function with respect to weights and biases

Main Method

X = np.array([[0,0,1], [0,1,1], [1,0,1], [1,1,1]])

y = np.array([[0],[1],[1],[0]])

nn = NeuralNetwork(X,y)

for i in range(1500):

nn.feedforward()

nn.backprop()

print(nn.output)

X1 | X2 | X3 | Y |

0 | 0 | 1 | 0 |

0 | 1 | 1 | 1 |

1 | 0 | 1 | 1 |

1 | 1 | 1 | 0 |

Testing

Prediction | Actual |

? | 0 |

? | 1 |

? | 1 |

? | 0 |

Results

Conclusion

- We have Successfully Built a Neural Network from Scratch
- Feed Forward
- Back Propagation
- Sigmoid Activation Function
- Future Investigations
- Convolutions (CNN)
- Learning Rate
- Different Activation Functions