APPLICATION OF SOFT COMPUTING 

Vinay Pratap Singh

​

ACTIVATION FUNCTION

• An Activation Function decides whether a neuron should be activated or not.
• It is a non-linear functions.
• An activation function is just a simple function that changes its inputs into outputs with a defined range.
• Activation Function also called as transfer functions are equations that define how the weighted sum of the input of a neural node is transformed into an output.

​

Activation for Hidden Layers

​

• Rectified Linear Activation (ReLU)
• Logistic (Sigmoid)
• Hyperbolic Tangent (Tanh)

​

Rectified Linear Activation (ReLU)

• The rectified linear activation function or ReLU is a non-linear function or piecewise linear function that will output the input directly if it is positive, otherwise, it will output zero.�It is the most commonly used activation function in neural networks, especially in Convolutional Neural Networks (CNNs) & Multilayer perceptrons.

Continue...

Program

def ReLU(x):�  if x>0:�    return x�  else: �    return 0

def relu(x):� return max(0.0, x)

Logistic (Sigmoid)

• It aims to predict the class to which a particular sample belongs
• The outcome is a discrete binary value, a probability between 0 and 1
• The model uses a function known as logistic function or sigmoid function and measures the relationship between dependent (outcome) and independent variables (features).

Continue...

Activation for Output Layers

• Y = tanh( X ) returns the hyperbolic tangent of the elements of X .
• The function takes any real value as input and outputs values in the range -1 to 1.
• The hyperbolic tangent activation function is also referred to simply as the Tanh (also “tanh” and “TanH“) function.
• It is very similar to the sigmoid activation function and even has the same S-shape. 

�Continue...

​

• Y = tanh( X )

�Activation for Output Layers

​

​

• Linear
• Logistic (Sigmoid)
• Softmax

​

• Softmax

​

• z is the vector of raw outputs from the neural network
• The value of e ≈ 2.718

​

Continue...

• The SoftMax function is used as the activation function in the output layer of neural network models that predict a multinomial probability

​

• SoftMax outputs between the range (0,1) so that the sum of the outputs is always 1. 

How to choose an output layer activation function

Binary ,Multiclass, Multilabel Classification

Single layer feed forward neural network

• Feed Forward Neural Network is an artificial neural network.
• the connections between nodes does not form a cycle. 
• The opposite of a feed forward neural network is a recurrent neural network, in which certain pathways are cycled. 
• Information is only processed in one direction.
• The feed forward model is the simplest form of neural network 
• the data may pass through multiple hidden nodes, it always moves in one direction and never backwards.

​

Continue... 

https://www.renom.jp/notebooks/tutorial/beginners_guide/feedforward_example_2/notebook.html

Multilayer feed forward neural network

• A multilayer feedforward neural network is an interconnection of perceptions in which data and calculations flow in a single direction, from the input data to the outputs.
• A Multi-Layer Perceptron (MLP) contains one or more hidden layers (apart from one input and one output layer). While a single layer perceptron can only learn linear functions, a multi-layer perceptron can also learn non - linear functions.

​

​

Disadvantage of FFNN

• They can't handle sequential data.
• Handle only current state but need previous state.
• Absent memory element. 

​

Recurrent neural network

• It has memory element.
• It can handle sequential data.
• Recurrent Neural Network(RNN) are a type of Neural Network where the output from previous step are fed as input to the current step.
• In traditional neural networks, all the inputs and outputs are independent of each other, but in cases like when it is required to predict the next word of a sentence, the previous words are required and hence there is a need to remember the previous words. 

Continue...

Ho = initial hidden value.

​

Continue...

• Calculate output

​

​

​

• Activation

​