Neural Networks from Scratch

An Algorithmic Approach

By: Ankit Gupta

(Founder/President @ DiscoverAI)

Components

  • Input Layer
  • Hidden Layer
  • Output Layer
  • Weights and Biases between Layers
  • Activation Function

Architecture

Neural Network Training

  • Output is designated by the following function:
    • Y = 𝜎(W2𝝈(W1x+b1)+b2)
      • Weights are represented by W1 and W2
      • Biases represented by b1 and b2
  • Two Steps in Training:
    • Feedforward
    • Backpropagation

Sequential Graph

Feedforward Implementation

def feedforward(self):

self.layer1 = sigmoid(np.dot(self.input, self.weights1))

self.output = sigmoid(np.dot(self.layer1, self.weights2))

Loss Function

  • Many Different Possibilities
  • We will choose: Sum of Squares Error =
  • Goal: Minimize the Loss Function

Backpropagation

  • Propagate the error back and update our weights and biases
  • Finds the extent to which we need to adjust the weights and biases

def backprop(self):

d_weights2 = np.dot(self.layer1.T, (2*(self.y - self.output) * sigmoid_derivative(self.output)))

d_weights1 = np.dot(self.input.T, (np.dot(2*(self.y - self.output) * sigmoid_derivative(self.output), self.weights2.T) * sigmoid_derivative(self.layer1)))

self.weights1 += d_weights1

self.weights2 += d_weights2

Gradient Descent

  • Derivative of the Loss Function with respect to weights and biases

Main Method

X = np.array([[0,0,1], [0,1,1], [1,0,1], [1,1,1]])

y = np.array([[0],[1],[1],[0]])

nn = NeuralNetwork(X,y)

for i in range(1500):

nn.feedforward()

nn.backprop()

print(nn.output)

X1

X2

X3

Y

0

0

1

0

0

1

1

1

1

0

1

1

1

1

1

0

Testing

Prediction

Actual

?

0

?

1

?

1

?

0

Results

Conclusion

  • We have Successfully Built a Neural Network from Scratch
    • Feed Forward
    • Back Propagation
    • Sigmoid Activation Function
  • Future Investigations
    • Convolutions (CNN)
    • Learning Rate
    • Different Activation Functions
Neural Networks from Scratch - Google Slides