1 of 69

Artificial Neural Networks�The Beginning of �Artificial Intelligence

​

Prof. Debasis Samanta

Indian Institute of Technology Kharagpur

2 of 69

  • Which of the part in Human is mightier than any part in any living entity?
    1. Leg
    2. Hand
    3. Tusk
    4. Brain

BDSET

2

DSamanta@IITKGP

3 of 69

Introduction to Artificial Intelligence

4 of 69

About AI

  • Digital Computer Revolution Began in 1950

BDSET

4

DSamanta@IITKGP

5 of 69

Von Neuman Architecture

  • Today’s computer

BDSET

5

DSamanta@IITKGP

6 of 69

Mark Minosi’s Dream

  • Computer: Machine versus Man

BDSET

6

DSamanta@IITKGP

7 of 69

Reality: Man versus Machine

  • Compuman

BDSET

7

DSamanta@IITKGP

8 of 69

Reality: Compuman

  • Processing speed 🡪 Computer

​

  • Memory 🡪 Computer

​

  • Human factors 🡪 Human
          • mind, mood, emotion, perception, cognition, psychology etc.
          • Breakthrough: Computer can behave very close to Human
          • Simon, Minasi etc.

BDSET

8

DSamanta@IITKGP

9 of 69

Towards the Reality….

  • So far computing speed is concerned enough has been achieved

​

  • More researches are in pipeline to achieve more

​

    • Quantum computing

​

    • Molecular electronics

​

    • DNA Computer

​

    • etc.

BDSET

9

DSamanta@IITKGP

10 of 69

​

?

BDSET

10

DSamanta@IITKGP

11 of 69

Towards AI

BDSET

11

DSamanta@IITKGP

12 of 69

History of Artificial Neural Networks

13 of 69

BDSET

13

DSamanta@IITKGP

Src: https://medium.com/analytics-vidhya/brief-history-of-neural-networks-44c2bf72eec

14 of 69

Why NN provides the state of art technique today but diminished in 80’s?

BDSET

14

DSamanta@IITKGP

    • Data was not sufficient to experiment
    • Computation power was limited
    • Thus, Theory lags dramatically no guarantees
    • Little understanding of limitations
    • Limited interpretability
    • Vanishing gradient problem

In early 80’s,

    • Google, Microsoft and other big data company provided large dataset
    • Large GPU power is available now
    • Good research and improved technique based on solid intuition
    • Though they’re not brand-new techniques
    • Now experiment can be done easily and fast to support theory

Today,

11

15 of 69

Concept of Artificial Neural Networks

16 of 69

BDSET

16

DSamanta@IITKGP

Brain: Centre of the nervous system

Gray matter

Cerebellum

Spinal Cord

17 of 69

BDSET

17

DSamanta@IITKGP

Biological nervous system

 

18 of 69

BDSET

18

DSamanta@IITKGP

Neuron: Basic unit of nervous system

19 of 69

BDSET

19

DSamanta@IITKGP

Neuron and its working

Figure shows a schematic of a biological neuron. There are different parts in it

  • Dendrite : A bush of very thin fibre.

​

  • Axon : A long cylindrical fibre.

​

  • Soma : It is also called a cell body, and just like as a nucleus of cell.

​

  • Synapse : It is a junction where axon makes contact with the dendrites of neighbouring dendrites.

20 of 69

BDSET

20

DSamanta@IITKGP

Neuron and its working

  • There is a chemical in each neuron called neurotransmitter.
  • A signal (also called sense) is transmitted across neurons by this chemical.
  • That is, all inputs from other neuron arrive to a neurons through dendrites.
  • These signals are accumulated at the synapse of the neuron and then serve as the output to be transmitted through the neuron.
  • An action may produce an electrical impulse, which usually lasts for about a millisecond.
  • Note that this pulse generated due to an incoming signal and all signal may not produce pulses in axon unless it crosses a threshold value.
  • Also, note that an action signal in axon of a neuron is commutative signals arrive at dendrites which summed up at soma.

21 of 69

BDSET

21

DSamanta@IITKGP

Neuron and its working

22 of 69

BDSET

22

DSamanta@IITKGP

Analogy between BNN and ANN

23 of 69

Neural Network

BDSET

23

DSamanta@IITKGP

24 of 69

BDSET

24

DSamanta@IITKGP

Artificial neural network

  • In fact, the human brain is a highly complex structure viewed as a massive, highly interconnected network of simple processing elements called neurons.

​

  • Artificial neural networks (ANNs) or simply we refer it as neural network (NNs), which are simplified models (i.e., imitations) of the biological nervous system, and obviously, therefore, have been motivated by the kind of computing performed by the human brain.

​

  • The behaviour of a biological neural network can be captured by a simple model called artificial neuron or perceptron.

25 of 69

BDSET

25

DSamanta@IITKGP

Artificial neural network

 

26 of 69

Analogy with Human Brain

HUMAN NEURON

PERCEPTRON

BDSET

26

DSamanta@IITKGP

MULTIPLE NERVE CELL

MULTI LAYER PERCEPTRON (MLP)

Src of Images: Internet

27 of 69

BDSET

27

DSamanta@IITKGP

Artificial neural network

We may note that a neutron is a part of an interconnected network of nervous system and serves the following.

  • Compute input signals
  • Transportation of signals (at a very high speed)
  • Storage of information
  • Perception, automatic training and learning

​

We also can see the analogy between the biological neuron and artificial neuron. Truly, every component of the model (i.e., artificial neuron) bears a direct analogy to that of a biological neuron. It is this model which forms the basis of neural network (i.e., artificial neural network).

28 of 69

BDSET

28

DSamanta@IITKGP

Artificial neural network

  • Note that, a biological neuron receives all inputs through the dendrites, sums them and produces an output if the sum is greater than a threshold value.

​

  • The input signals are passed on to the cell body through the synapse, which may accelerate or retard an arriving signal.

​

  • It is this acceleration or retardation of the input signals that is modelled by the weights.

​

  • An effective synapse, which transmits a stronger signal will have a correspondingly larger weights while a weak synapse will have smaller weights.

​

  • Thus, weights here are multiplicative factors of the inputs to account for the strength of the synapse.

29 of 69

BDSET

29

DSamanta@IITKGP

Artificial neural network

 

 

30 of 69

Activation Functions

31 of 69

BDSET

31

DSamanta@IITKGP

Artificial neural network

 

32 of 69

BDSET

32

DSamanta@IITKGP

Transformation functions

 

33 of 69

BDSET

33

DSamanta@IITKGP

Transformation functions

34 of 69

BDSET

34

DSamanta@IITKGP

Transformation functions

 

35 of 69

BDSET

35

DSamanta@IITKGP

Transfer functions in ANN

36 of 69

Transformation functions

BDSET

36

DSamanta@IITKGP

Sigmoid Activation Function

Linear Activation Function

37 of 69

Transformation functions

BDSET

37

DSamanta@IITKGP

ReLU Activation Function

Leaky ReLU Activation Function

38 of 69

Transformation functions

BDSET

38

DSamanta@IITKGP

Hyperbolic Tangent Activation Function

Softmax Activation Function

39 of 69

Transformation functions

BDSET

39

DSamanta@IITKGP

40 of 69

Neural Network Computing

BDSET

40

DSamanta@IITKGP

This diagram shown is a model of a single neural unit called perceptron.

A set of synapses or connecting links, each of which is characterized by it’s own weights.

An Adder for summing the input signals, weighted by the respective synaptic weight.

An activation function for limiting the amplitude of the output of a neuron. (Also called squashing function)

Multiple such neuron structures are linked together to make a neural system called Multi-Layer Perceptron (MLP).

The output of one neuron is input to the other neuron in next layer only.

No neuron in the same layer can be connected.

∑

ⱷ(.)

W1k

W2k

W3k

Wmk

.

.

.

.

.

.

Ѵk

X1

X2

X3

Xm

bias

bk

Synaptic

Weights

Summing

Junction

Activation

Function

Input

Output

ŷk

Node k

ŷk

ŷk

 

 

Given X = {x1, x2, .. xm}

41 of 69

Neural Network Computing

BDSET

41

DSamanta@IITKGP

  • Input layer (source nodes) takes input / variable.
  • There can be one or more hidden layer and one output layer.
  • For multi class classification problem, output layer has multiple units and for binary class classification problem has a single binany output unit .
  • By adding more hidden layers, network is enabled to extract higher order statistics from it’s input.
  • Another names of neural network are Multi layer Perceptron or Artificial Neural Network or Multilayer feedforward Network.
  • If every node in layer l is connected to every node of layer l+1, then such network is called fully connected network otherwise partially connected network.

42 of 69

BDSET

42

DSamanta@IITKGP

Advantages of ANN

  • ANNs exhibits mapping capabilities, that is, they can map input patterns to their associated output pattern.

​

  • The ANNs learn by examples. Thus, an ANN architecture can be trained with known example of a problem before they are tested for their inference capabilities on unknown instance of the problem. In other words, they can identify new objects previously untrained.

​

  • The ANNs posses the capability to generalize. This is the power to apply in application where exact mathematical model to problem are not possible.

43 of 69

BDSET

43

DSamanta@IITKGP

Advantages of ANN

  • The ANNs are robust system and fault tolerant. They can therefore, recall full patterns from incomplete, partial or noisy patterns.

​

  • The ANNS can process information in parallel, at high speed and in a distributed manner.

​

  • Thus a massively parallel distributed processing system made up of highly interconnected (artificial) neural computing elements having ability to learn and acquire knowledge is possible.

44 of 69

Applications of ANNs

BDSET

44

DSamanta@IITKGP

Pattern Classification

Medical Application

Forecasting

Anomaly / Outlier Detection

Adaptive Filtering

Adaptive Control

45 of 69

Why ANNs?

BDSET

45

DSamanta@IITKGP

It’s power to solve complex real world case study.

    • Image Processing / Computer Vision
    • Audio or Video Processing
    • Text Translation, Word Prediction etc.

Real world problem has large no of complex features which is hard to solve by classical ML algorithms.

Like human brain can do complex tasks through various neuron networks, Deep Learning also try to mimic human brain.

Let’s understand each of these points one by one.

46 of 69

ANN Architectures

47 of 69

BDSET

47

DSamanta@IITKGP

Neural network architectures

There are three fundamental classes of ANN architectures:

    • Single layer feed forward architecture
    • Multilayer feed forward architecture
    • Recurrent networks architecture

​

Before going to discuss all these architectures, we first discuss the mathematical details of a neuron at a single level. To do this, let us first consider the AND problem and its possible solution with neural network.

48 of 69

BDSET

48

DSamanta@IITKGP

The AND problem and its Neural network

 

49 of 69

BDSET

49

DSamanta@IITKGP

The AND problem and its Neural network

Alternatively, the AND problem can be thought as a perception problem where we have to receive four different patterns as input and perceive the results as 0 or 1.

50 of 69

BDSET

50

DSamanta@IITKGP

The AND problem and its Neural network

A possible neuron specification to solve the AND problem is given in the following. In this solution, when the input is 11, the weight sum exceeds the threshold ( = 0.9) leading to the output 1 else it gives the output 0.

 

51 of 69

BDSET

51

DSamanta@IITKGP

Single layer feed forward neural network

The concept of the AND problem and its solution with a single neuron can be extended to multiple neurons.

52 of 69

BDSET

52

DSamanta@IITKGP

Single layer feed forward neural network

  • We see, a layer of n neurons constitutes a single layer feed forward neural network.

​

  • This is so called because, it contains a single layer of artificial neurons.

​

  • Note that the input layer and output layer, which receive input signals and transmit output signals are although called layers, they are actually boundary of the architecture and hence truly not layers.

​

  • The only layer in the architecture is the synaptic links carrying the weights connect every input to the output neurons.

53 of 69

BDSET

53

DSamanta@IITKGP

Modeling SLFFNN

 

54 of 69

BDSET

54

DSamanta@IITKGP

Multilayer feed forward neural networks

 

55 of 69

BDSET

55

DSamanta@IITKGP

Multilayer feed forward neural networks

 

56 of 69

BDSET

56

DSamanta@IITKGP

Multilayer feed forward neural networks

 

57 of 69

BDSET

57

DSamanta@IITKGP

Recurrent neural network architecture

  • The networks differ from feedback network architectures in the sense that there is at least one “feedback loop”.

​

  • Thus, in these networks, there could exist one layer with feedback connection.

​

  • There could also be neurons with self-feedback links, that is, the output of a neuron is fed back into itself as input.

58 of 69

BDSET

58

DSamanta@IITKGP

Recurrent neural network architecture

Depending on different type of feedback loops, several recurrent neural networks are known such as Hopfield network, Boltzmann machine network etc.

59 of 69

BDSET

59

DSamanta@IITKGP

Why different type of neural network architectures?

To give the answer to this question, let us first consider the case of a single neural network with two inputs as shown below.

60 of 69

BDSET

60

DSamanta@IITKGP

Revisit of a single neural network

 

 

61 of 69

BDSET

61

DSamanta@IITKGP

AND problem is linearly separable

62 of 69

BDSET

62

DSamanta@IITKGP

XOR problem is linearly non-separable

63 of 69

BDSET

63

DSamanta@IITKGP

Our observations

  • From the example discussed, we understand that a straight line is possible in AND-problem to separate two tasks namely the output as 0 or 1 for any input.
  • However, in case of XOR problem, such a line is not possible.

Note:

Horizontal or a vertical line in case of XOR problem is not admissible because in that case it completely ignores one input.

64 of 69

BDSET

64

DSamanta@IITKGP

Example

 

65 of 69

BDSET

65

DSamanta@IITKGP

Example: Solving XOR problem

66 of 69

BDSET

66

DSamanta@IITKGP

Dynamic neural network

  • In some cases, the output needs to be compared with its target values to determine an error, if any.

​

  • Based on this category of applications, a neural network can be static neural network or dynamic neural network.

​

  • In a static neural network, error in prediction is neither calculated nor feedback for updating the neural network.

​

  • On the other hand, in a dynamic neural network, the error is determined and then feed back to the network to modify its weights (or architecture or both).

67 of 69

BDSET

67

DSamanta@IITKGP

Dynamic neural network

68 of 69

BDSET

68

DSamanta@IITKGP

Dynamic neural network

From the above discussions, we conclude that

​

​

    • For linearly separable problems, we solve using single layer feed forward neural network.

​

​

    • For non-linearly separable problem, we solve using multilayer feed forward neural networks.

​

​

    • For problems, with error calculation, we solve using recurrent neural networks as well as dynamic neural networks.

69 of 69

BDSET

69

DSamanta@IITKGP

Thank You!!