Intro to Neural Networks

By: Sid

What are Neural Networks?

• Neural networks are a type of machine learning structure that is modeled after the human brain
• Neural networks can be used for things like: housing price prediction, Ad placement and suggestions, Photo tagging, Speech recognition, Autonomous Driving
• Types of Neural Networks:
• Convolutional NN (for images and photo tagging)
• Radial NN
• Recurrent NN
• Standard NN

Basic Types of Learning

• Supervised Learning - the network generates predicted outputs, then a “teacher” provides the correct answers to the examples; they are compared and the network adjusts itself based on the correct answer
• Unsupervised Learning - the network identifies patterns in a dataset without any external influence (this is most commonly used in cluster analysis)

​

Standard Neural Networks

The input layer takes in the various factors that can affect the output (ex. weather: air pressure, temperature, wind speed, location)

​

​

When it gets complicated...

For areas like Finance, there can be thousands of inputs that complicate networks; this is why the predictions take lots of computing power (leads to a nifty field called Big Data Analytics)

Review of Biology - Neurons, Dendrites & Axons

Many electrical signals move through the dendrites into the cell body. If the net electrical signal crosses a certain threshold, the signal is passed on to another Neuron.

The Perceptron Model

The activation function f(x) is usually the sigmoid function:

​

​

​

Or the ReLU function:

​

​

​

​

​

b is the bias that is added to each node

​

The Perceptron Model

This is the same model. It just shows things more clearly.

​

Note: ŷ is the output of the network...it is only a predicted value. The correct answer provided by the teacher is called y. - this is supervised learning

​

Some Common Notations

• Training set = {(x⁽¹⁾, y⁽¹⁾), (x⁽²⁾, y⁽²⁾), (x⁽³⁾, y⁽³⁾) …., (x^(m), y^(m))} - x is usually an array of inputs
• More training examples (m) -> more accurate outputs
• ŷ - mentioned before; this is the predicted value while y is the correct value

Logistic Regression Cost Function

• Given a training set, we want ŷ⁽ⁱ⁾ ≈ y
• Loss (error) function: L(ŷ, y) - this is for each training example
• usually, L(ŷ, y) = ½ (ŷ - y)²
• In this case, L(ŷ, y) = -(y lnŷ + (1-y) ln(1-ŷ)) - why? It makes it more accurate that the regular one
• Cost Function: J(w, b) = 1/m ∑ L(ŷ⁽ⁱ⁾, y⁽ⁱ⁾) - this is what we want to minimize

Gradient Descent

• To adjust weights and biases, repeat:
• w_new = w_old - ⍺ ∂J(w,b)/∂w
• b_new = b_old - ⍺ ∂J(w,b)/∂b
• ⍺ - is called the learning rate; it multiplies the cost function, which determines how much it changes with each iteration (if it is too high, the weights will be over adjusted and produce inaccurate ŷ)

​

​

Some extra stuff for you...

• OpenNN
• R Language
• TensorFlow - Python, Javascript
• Tutorials in the resources folders

​