Artificial Intelligence and Machine Learning

Multi Layer Neural Network

Week 4 - Lecture

The Perceptron Model.

• The Perceptron, introduced by Frank Rosenblatt and later analysed by Minsky and Papert, extends the McCulloch-Pitts neuron
• by incorporating learnable numerical weights and
• an adaptive learning process.
• It serves as a Linear Binary Classifier, dividing data into two categories based on a linear decision boundary.

​

​

​

Week - 4 - Multi Layer Neural Network

2

Perceptron for Real Value Inputs.

• The step function is unsuitable for gradient-based learning: it is non-differentiable at 0, has zero gradient everywhere else, and gives only a binary output.
• The step function may not capture the subtleties of the data;
• it creates a sharp boundary at 0,
• meaning that all positive sums are classified as class 1
• and all negative sums are classified as class 0.

3

The activation function

• In General, Activation Functions are group of function that introduces non-linearity to the output of neurons.
• The activation function does the non-linear transformation to the input, making it capable to learn and perform more complex tasks.

​

​

​

​

​

​

4

Importance of Activation Functions

The purpose of the activation functions is to introduce non linearity to the neural network

What if we wanted to build the neural network to distinguish these two different type of data points?

Importance of Activation Functions

The purpose of the activation functions is to introduce non linearity to the neural network

Activation functions allow us to approximate such complex non linear functions

Some common Activation Functions

Threshold Function

Linear Function

Sigmoid Functions

ReLU Function

The Perceptron :Forward Propagation

Building Neural Network with Perceptrons

Limitations of Perceptron

• Could not classify the non linearly separable data
• But we also realized that if we arrange multiple perceptrons in layers it can solve such nonlinear problems

Neural Network and The Theory of Universal Approximation

• A network with one hidden layer, a non-linear activation and enough neurons can approximate any continuous function (on a bounded input range) to any accuracy.

​

• How many neurons?

​

For Boolean functions of n inputs, at most 2^n neurons in one hidden layer suffice

​

e.g. for n = 2 inputs:

2^2 = 4 neurons in one hidden layer

​

​

Single Layer Neural Network

As every node in a layer is connected to every node of the previous layer, it is called a Dense (fully connected) Layer

Challenges of one hidden layer network

2^n neurons may work well for Boolean input but does not work the same for real world problems

​

As we may need millions of neurons in the hidden layer which is computationally inefficient!!!

(eg. Dog vs Cat classification)

​

Solution: Using deeper neural networks aka Multi Layered Neural Network!!

Deep Neural Networks

Why go deeper?

​

• Inherently compositional - builds complex patterns from simple ones

​

• Can be more expressive for the same parameter count

​

• Deep networks efficiently represent hierarchical structures found in real-world data

Deep Neural Network Architecture

How DNN work! (An example)

image from internet, subjected to copyrights!

How DNN work! (An example)

image from internet, subjected to copyrights!

Quantifying Loss (Loss Function)

• The loss function measures the discrepancy between the model's predictions and the actual values.

​

​

Predicted

Actual

Empirical Loss (Cost)

Measures the average loss over the entire dataset:

Also known as :

Objective Function /

Cost Function /

Empirical Risk

Choosing the right Cost function

Empirical Risk Minimization

• We want to find the network weights that achieve the lowest cost

Empirical Risk Minimization

The Cost is the function of network weights w/θ

​

Our Goal is to minimize it

Empirical Risk Minimization

How?

​

Remember Gradient Descent?

​

Empirical Risk Minimization

Gradient Descent Algorithm

Computing Gradient in DNN

Computing Gradient in DNN

X

Apply Chain Rule Here

Computing Gradient in DNN

W₂

Computing Gradient in DNN

W₂

Repeat this for the weights of every layers using gradient from later layers!

Back propagation Algorithm

​

The training proceeds in two phases:

• Forward Pass
• Backward Pass

​

Forward Pass:

• In this phase, the synaptic weights of the network are fixed and the input signal is propagated through the network, layer by layer, until it reaches the output.

Backward Pass

In this phase, an error signal is produced by comparing the output of the network with a desired response.

​

The resulting error signal is propagated in backward direction through the network, again layer by layer.

​

In this second phase, successive adjustments are made to the synaptic weights of the network

​

Backpropagation Algorithm

​

​

• The error signal at the output of neuron j at iteration n is given by

32

Backpropagation Algorithm

The total cost E(n) for all the neurons in the output layer is therefore

33

Backpropagation Algorithm

• Let N be the total number of training vectors (examples). Then the average squared error is:

​

​

• Consider the neuron j ,The input signal v_j(n) and output y_j(n) of neuron j is given by:

34

Back propagation Algorithm

Backpropagation Algorithm

• The correction made to the weight is proportional to the partial derivative of instantaneous error
• Using the chain rule of calculus, this gradient can be expressed as follows:

​

​

​

• We can get following partial derivatives

​

36

Backpropagation Algorithm

​

​

​

• Putting all partial derivatives in equation 1, we get

​

​

• The correction applied to the weight is defined by

37

Backpropagation Algorithm

• Using equation (2), equation (3) can be written as

​

​

• This can be written as

​

​

​

• The error term in equation 4 and 5 depends upon location of neuron in the MLP. There can be two possible scenarios.

​

​

​

​

38

Backpropagation Algorithm

Case I: Neuron j is output layer neuron

• The desired response d_j(n) for the neuron j is directly available. Computation of the error e_j(n) is straightforward in this case. We can use equation 4 or 5 to calculate weight update term.

​

​

​

39

where:

​

​

​

Therefore :

​

Backpropagation Algorithm

​

Case II: Neuron j is hidden layer neuron

• There is no desired response available for neuron j. The error signal for a hidden neuron must be determined recursively in terms of the error signals of all neurons connected to it as below:

​

​

40

Backpropagation Algorithm

Algorithm

1. Initialize all weights and biases in network
2. While terminating condition is not satisfied
3. for each training tuple X in D
4. for each input layer unit j: y_j = x_j; // output of an input unit is its actual input value
5. for each hidden or output layer unit j

compute the net input of unit j with respect to the previous

layer,

compute the output of each unit j

​

41

Learning

Algorithm

6. for each unit j in the output layer

​

7. for each unit j in the hidden layers, from the last to the first hidden layer

​

​

8. for each weight w_ji in network
9. ​

​

//Weight Update

​

​

42

Gradient Descent in Practice

• Based on how much data we use to compute the gradient of the objective function, there are three main variants of gradient descent:
• Batch Gradient Descent.
• Stochastic Gradient Descent.
• Mini-Batch Gradient Descent.

​

Batch Gradient Descent

Stochastic Gradient Descent

Mini- Batch Gradient Descent

Challenges

• Same learning rate applies to all parameter updates.
• Getting trapped into suboptimal local minima or saddle points.
• Choosing a Proper Learning Rate.
• Annealing:
• Learning rate schedules or reducing the learning rate according to a pre-defined schedule or when the change in the objective between epochs falls below a threshold.
• Schedules and threshold values have to be pre-defined, thus may not be able to adapt to a dataset’s characteristics.

Thank You!