Backpropogation in Neural Networks

​

RAJKUMAR D

ASST PROG(SL.G)

DEPARTMENT OF COMPUTER SCIENCE & APPLICATIONS

SRMIST, RAMAPURAM

Backpropogation- Introduction

• Backpropagation is the essence of neural network training. It is the method of fine-tuning the weights of a neural network based on the error rate obtained in the previous epoch (i.e., iteration).
• Proper tuning of the weights allows you to reduce error rates and make the model reliable by increasing its generalization.
• Backpropagation in neural network is a short form for “backward propagation of errors.” It is a standard method of training artificial neural networks.
• This method helps calculate the gradient of a loss function with respect to all the weights in the network.

How Backpropagation Algorithm Works

• The Back propagation algorithm in neural network computes the gradient of the loss function for a single weight by the chain rule.
• It efficiently computes one layer at a time, unlike a native direct computation.
• It computes the gradient, but it does not define how the gradient is used. It generalizes the computation in the delta rule.

How Backpropagation Algorithm Works

How Backpropagation Algorithm Works

• Inputs X, arrive through the preconnected path
• Input is modeled using real weights W. The weights are usually randomly selected.
• Calculate the output for every neuron from the input layer, to the hidden layers, to the output layer.
• Calculate the error in the outputs

ErrorB= Actual Output – Desired Output

• Travel back from the output layer to the hidden layer to adjust the weights such that the error is decreased.
• Keep repeating the process until the desired output is achieved

​

Why We Need Backpropagation?

Most prominent advantages of Backpropagation are:

• Backpropagation is fast, simple and easy to program
• It has no parameters to tune apart from the numbers of input
• It is a flexible method as it does not require prior knowledge about the network
• It is a standard method that generally works well
• It does not need any special mention of the features of the function to be learned.

​

Artificial Neural Network

Comparison of ANN and Bio Neural Network

• Dendrites from Biological Neural Network represent inputs in Artificial Neural Networks, cell nucleus represents Nodes, synapse represents Weights, and Axon represents Output.

​

��Architecture of an Artificial Neural Network

• To understand the concept of the architecture of an artificial neural network, we have to understand what a neural network consists of. In order to define a neural network that consists of a large number of artificial neurons, which are termed units arranged in a sequence of layers.
• Lets us look at various types of layers available in an artificial neural network.

​

��Architecture of an Artificial Neural Network

Input Layer:

• As the name suggests, it accepts inputs in several different formats provided by the programmer.

Hidden Layer:

• The hidden layer presents in-between input and output layers. It performs all the calculations to find hidden features and patterns.

Output Layer:

• The input goes through a series of transformations using the hidden layer, which finally results in output that is conveyed using this layer.

​

​

��Architecture of an Artificial Neural Network

• The artificial neural network takes input and computes the weighted sum of the inputs and includes a bias. This computation is represented in the form of a transfer function.

​

​

• It determines weighted total is passed as an input to an activation function to produce the output. Activation functions choose whether a node should fire or not.
• Only those who are fired make it to the output layer. There are distinctive activation functions available that can be applied upon the sort of task we are performing.

​

Advantages of Artificial Neural Network (ANN)

1. Parallel processing capability:

• Artificial neural networks have a numerical value that can perform more than one task simultaneously.

2. Storing data on the entire network:

• Data that is used in traditional programming is stored on the whole network, not on a database.
• The disappearance of a couple of pieces of data in one place doesn't prevent the network from working.

3. Capability to work with incomplete knowledge:

• After ANN training, the information may produce output even with inadequate data.
• The loss of performance here relies upon the significance of missing data.

​

Advantages of Artificial Neural Network (ANN)

4. Having a memory distribution:

• For ANN is to be able to adapt, it is important to determine the examples and to encourage the network according to the desired output by demonstrating these examples to the network.
• The succession of the network is directly proportional to the chosen instances, and if the event can't appear to the network in all its aspects, it can produce false output.

5. Having fault tolerance:

• Extortion of one or more cells of ANN does not prohibit it from generating output, and this feature makes the network fault-tolerance.

​

Disdvantages of Artificial Neural Network (ANN)

1. Assurance of proper network structure:

• There is no particular guideline for determining the structure of artificial neural networks.
• The appropriate network structure is accomplished through experience, trial, and error.

2. Unrecognized behavior of the network:

• It is the most significant issue of ANN. When ANN produces a testing solution, it does not provide insight concerning why and how. It decreases trust in the network.

3. Hardware dependence:

• Artificial neural networks need processors with parallel processing power, as per their structure. Therefore, the realization of the equipment is dependent.

​

Disdvantages of Artificial Neural Network (ANN)

4. Difficulty of showing the issue to the network:

• ANNs can work with numerical data. Problems must be converted into numerical values before being introduced to ANN.
• The presentation mechanism to be resolved here will directly impact the performance of the network. It relies on the user's abilities.

5. The duration of the network is unknown:

• The network is reduced to a specific value of the error, and this value does not give us optimum results.

​

How do artificial neural networks work?

• Artificial Neural Network can be best represented as a weighted directed graph, where the artificial neurons form the nodes.
• The association between the neurons outputs and neuron inputs can be viewed as the directed edges with weights.
• The Artificial Neural Network receives the input signal from the external source in the form of a pattern and image in the form of a vector.
• These inputs are then mathematically assigned by the notations x(n) for every n number of inputs.

How do artificial neural networks work?

How do artificial neural networks work?

• Afterward, each of the input is multiplied by its corresponding weights ( these weights are the details utilized by the artificial neural networks to solve a specific problem ).
• In general terms, these weights normally represent the strength of the interconnection between neurons inside the artificial neural network.
• All the weighted inputs are summarized inside the computing unit.
• If the weighted sum is equal to zero, then bias is added to make the output non-zero or something else to scale up to the system's response.
• Bias has the same input, and weight equals to 1.
• Here the total of weighted inputs can be in the range of 0 to positive infinity. Here, to keep the response in the limits of the desired value, a certain maximum value is benchmarked, and the total of weighted inputs is passed through the activation function.

How do artificial neural networks work?

• The activation function refers to the set of transfer functions used to achieve the desired output.
• There is a different kind of the activation function, but primarily either linear or non-linear sets of functions.
• Some of the commonly used sets of activation functions are the Binary, linear, and Tan hyperbolic sigmoidal activation functions. Let us take a look at each of them in details:

1. Binary:

• In binary activation function, the output is either a one or a 0. Here, to accomplish this, there is a threshold value set up. If the net weighted input of neurons is more than 1, then the final output of the activation function is returned as one or else the output is returned as 0.

​

How do artificial neural networks work?

2. Sigmoidal Hyperbolic:

• The Sigmoidal Hyperbola function is generally seen as an "S" shaped curve.
• Here the tan hyperbolic function is used to approximate output from the actual net input.
• The function is defined as:

F(x) = (1/1 + exp(-????x))

Where ???? is considered the Steepness parameter.

​

Applications of Neural Network:

Aerospace:Aircraft component fault detectors and simulations, aircraft control systems, high-performance auto-piloting, and flight path simulations.

Automotive:Improved guidance systems, development of power trains, virtual sensors, and warranty activity analyzers.

Electronics: Chip failure analysis, circuit chip layouts, machine vision, non-linear modeling, prediction of the code sequence, process control, and voice synthesis.

�