Getting Started with Deep Learning and Neural Network

Week - 3 - Getting Started with Deep Learning and Neural Network

1

6CS012 – Artificial Intelligence and Machine Learning.�Week – 3

Learning Objective,

• Discuss on the limitations of Logistic Regression for image classification Task?

​

• A brief history of neural networks.

​

• Early Models and their limitations.

​

• Introduction to Modern Neural Networks.

​

Week - 3 - Getting Started with Deep Learning and Neural Network

2

Machine 🡪 Deep Learning?

Why?

Week - 3 - Getting Started with Deep Learning and Neural Network

3

Last week : Logistic Regression for Classification.

• We implemented Logistic Regression using ERM Framework:

Week - 3 - Getting Started with Deep Learning and Neural Network

4

Last week : Dissecting Logistic Regression.

Week - 3 - Getting Started with Deep Learning and Neural Network

5

Last week : Application.

• We use Softmax Regression and Build:
• “MNIST Handwritten Digit Classification.”

Week - 3 - Getting Started with Deep Learning and Neural Network

6

Our Observation – 1:

• Non-Linear Decision Boundary:
• Logistic Regression{Softmax Regression} was better on separating a data in classes/label those were linearly separable.
• What for data where classes/label are not linearly separable?
• {In ML there exists more advance algorithm for example DT, SVM etc. but not suitable for unstructured dataset like images.}

​

Week - 3 - Getting Started with Deep Learning and Neural Network

7

Our Observation – 2:

Week - 3 - Getting Started with Deep Learning and Neural Network

8

Why Feature extraction is a Challenge?

Machine Learning Vs. Deep Learning.

• Machine Learning:
• One crucial aspect of machine learning approaches to solving problems is that human and often undervalued engineering plays an important role.
• A human still has to frame the problem,
• acquire and organize data, design a space of possible solutions,
• select a learning algorithm and its parameters,
• apply the algorithm to the data,
• validate the resulting solution to decide whether it’s good enough to use, etc.
• These steps are of great importance.

​

Week - 3 - Getting Started with Deep Learning and Neural Network

9

Why Feature Extraction is a Limitations?

• Manual Effort & Expertise Required:
• Traditional methods rely on handcrafted features such as edge detectors , texture descriptors , or color histograms.
• Engineers and researchers must experiment with different feature extraction techniques to find the best ones for specific tasks.
• Suboptimal Performance:
• Handcrafted features may not always capture the most relevant information in complex images.
• They are often designed for specific datasets and may not generalize well across different conditions (e.g., lighting variations, occlusions).
• Curse of Dimensionality:
• Extracted features can be high-dimensional, making it computationally expensive to process and requiring dimensionality reduction techniques like PCA.
• Irrelevant or redundant features can degrade model performance.
• Lack of Adaptability:
• Features designed for one problem (e.g., object classification) might not be suitable for another (e.g., object detection or segmentation).
• Traditional feature extraction does not automatically adapt to variations in image quality, background, or perspective.

​

​

​

​

Week - 3 - Getting Started with Deep Learning and Neural Network

10

Machine Learning Vs. Deep Learning.

• Can we learn the underlying features directly from data?
• Yes, Deep Learning.

{ We will talk more on this as we move forward with more complex deep learning algorithms.}

Week - 3 - Getting Started with Deep Learning and Neural Network

11

Modern Breakthrough of Deep Learning.

• Deep learning have become one of the main approaches to AI
• They have been successfully applied to various pattern recognition, prediction, and analysis problems
• In many problems they have established the state of the art
• Often exceeding previous benchmarks by large margins
• Sometimes solving problems, you couldn’t solve using earlier ML methods

​

Week - 3 - Getting Started with Deep Learning and Neural Network

12

The Common Elements in above Breakthrough.

Week - 3 - Getting Started with Deep Learning and Neural Network

13

Neural Networks are Taking Over.

Week - 3 - Getting Started with Deep Learning and Neural Network

14

What are Neural Networks?

It is said to be inspired by “human ability of thinking”.

Introduction to Neural Networks.

From Biological Inspiration to Computational Models.

Week - 3 - Getting Started with Deep Learning and Neural Network

15

Way before Neural network …

• Humans have the remarkable ability to:
• Learn and Solve problems
• Recognize patterns and Memorize
• Create
• Think deeply

​

• but how exactly do humans function or think?
• Can we emulate i.e. put into machine?

​

Week - 3 - Getting Started with Deep Learning and Neural Network

16

Early Models of Human Cognition.

​

​

​

​

• Associationism (– 400 BC – 1900 AD by Aristotle, Plato, David Hume, Ivan Pavlov, …)
• 360 BC: Aristotle wrote:
• "Hence, too, it is that we hunt through the mental train, excogitating from the present or some other, and from similar or contrary or coadjacent. Through this process reminiscence takes place. For the movements are, in these cases, sometimes at the same time, sometimes parts of the same whole, so that the subsequent movement is already more than half accomplished.”
• In English: We memorize and rationalize through association.
• Theory of Associationism:
• “Learning is a mental process that forms associations between temporally related phenomena.”
• Example: “hey here’s a bolt of lightning, we’re going to hear thunder”
• Challenge: But where are the associations stored and how?

Week - 3 - Getting Started with Deep Learning and Neural Network

17

Beginning the Era of Connectionism.

• {Cautions: we have lots of historical development that happened and will discuss major events that happened in Mid 1800s.}
• Alexander Bain, in 1873 in his work “Mind and Body” proposed that
• “The information is in the connections” suggesting:
• The brain is a mass of interconnected neurons.
• He floated the idea: The brain is a network of neurons, and
• collection of neurons excite and stimulate each other, and memories are stored in the connections.
• different combinations of inputs can result in different outputs.

Week - 3 - Getting Started with Deep Learning and Neural Network

18

Connectionist Machines

• Neural networks are connectionist machines as opposed to Von – Neumann Machines.
• The machine has many non-linear processing units
• The program is the connections between these units
• Connections may also define memory

​

​

Week - 3 - Getting Started with Deep Learning and Neural Network

19

Alan Turing’s Connectionist Model - 1948

But what are these independent processing units?

Connectionist Machines: Warning

• Current deep learning algorithms emulate neural networks but run on
• Von Neumann machines, which have a fundamentally different architecture from biological brains.
• This creates both strengths and limitations in modern AI systems.

Week - 3 - Getting Started with Deep Learning and Neural Network

20

Modelling the Brain: Neurons.

• What are the units?
• Answer: A neuron.
• The structure of Biological Neurons {major elements}:
• Dendrite: receives signals from other neurons
• Synapse: point of connection to other neurons
• Soma: process the information
• Axon: transmits the output of this neuron
• How neurons transmits the information?
• There are roughly 86 billion neurons in the human brain.

​

Week - 3 - Getting Started with Deep Learning and Neural Network

21

Modelling the Brain: Connectionist.

22

Biological Neurons to Computational Model.

• Observations from working of Neurons:
• Billions of interconnected neurons (each linked to thousands of others) work in parallel
• For neurons to activate or fire or spike certain threshold must be passed
• Inputs are received and can be of:
• Excitatory inputs/Synapses:
• Transmits input to the neuron.
• Inhibitory inputs/Synapses:
• Any signals from an inhibitory inputs prevents neuron from firing, regardless of other inputs.

​

Week - 3 - Getting Started with Deep Learning and Neural Network

23

Biological Neurons to Computational Model.

• Neurons as Biological Computational devices:
• How can we model the single neuron based on the above observation?
• A Neuron can either fire or not-fire,
• thus, output of neuron can be binary i.e. 0 or 1
• As output can only be 0 or 1,
• if we can make collection of input also be binary,
• we can create a threshold function using a logic operator.
• Our First computational model was based on above thought and
• was called McCulloch & Pitts Neuron.

Week - 3 - Getting Started with Deep Learning and Neural Network

24

Fig: A Symbolic representation of “How we want our Neuron to be?”

McCulloch and Pitts Neuron.

{Earlier Computational or Mathematical Representations of Neuron.}

Week - 3 - Getting Started with Deep Learning and Neural Network

25

The McCulloch – Pitts Artificial Neuron.

• The first computational model of a neuron was proposed by Warren McCulloch (neuroscientist) and Walter Pitts (logician) in 1943 in their paper titled:
• “A Logical Calculus of the Ideas Immanent in Nervous Activity (1943).”
• Based on their thought and previous work on neural nets:
• on what was the fundamentals elements to represent computation in biological neurons
• they proposed a model also known as
• linear threshold gate or threshold logic units or MCP neurons.
• Modeled the neurons of the brain (and the brain itself) as performing propositional logic, where each neuron evaluates the truth value of its input (propositions)
• Effectively Boolean logic aka synaptic model

Week - 3 - Getting Started with Deep Learning and Neural Network

26

An MCP Neuron.

• Excitatory synapse:
• Transmits weighted input to the neuron
• Inhibitory synapse:
• Any signal from an inhibitory synapse prevents neuron from firing
• The activity of any inhibitory synapse absolutely prevents excitation of the neuron at that time.
• Regardless of other inputs

Week - 3 - Getting Started with Deep Learning and Neural Network

27

Computational Model of MCP Neuron.

• Mathematical Formalization of MCP.
• MCP describes the activity of single neuron with two states: firing(1) or not firing(0).

​

Week - 3 - Getting Started with Deep Learning and Neural Network

28

MCP Neuron and Boolean Operations.

• For emulating AND Functions:
• The AND function is "activated" only when all the incoming inputs are "on",
• this is, it outputs a 1 only when all inputs are 1.
• What will be my Threshold Function?

​

Week - 3 - Getting Started with Deep Learning and Neural Network

29

MCP Neuron and Boolean Operations.

• For OR Function:
• The OR function is "activated" when at least one of the incoming inputs is "on".
• What will be my Threshold Function?

​

Week - 3 - Getting Started with Deep Learning and Neural Network

30

MCP Neuron and Boolean Operations.

• For XOR Function:
• The XOR (Exclusive OR) function is activated (outputs "on" or 1)
• when exactly one of the incoming inputs is "on" (1), but not both.
• What will be my Threshold Function?

​

​

​

​

​

​

​

​

​

• The phenomenon also called “XOR Problem” or “XOR Realizations”.

Week - 3 - Getting Started with Deep Learning and Neural Network

31

Limitation of MCP Neurons.

• Limited to linearly separable Boolean functions
• Single neuron architecture (no complex connectivity patterns)
• Cannot process continuous values
• Lacks measure of importance (weights)
• No learning procedure
• Simplistic model compared to biological networks

​

Week - 3 - Getting Started with Deep Learning and Neural Network

32

Towards a Better Model.

{Introduction to Perceptron and Perceptron Learning Theory.}

Week - 3 - Getting Started with Deep Learning and Neural Network

33

The Perceptron Model.

• The Perceptron, introduced by Frank Rosenblatt and later analysed by Minsky and Papert, extends the McCulloch-Pitts neuron
• by incorporating learnable numerical weights and
• an adaptive learning process.
• It serves as a Linear Binary Classifier, dividing data into two categories based on a linear decision boundary.

​

​

​

Week - 3 - Getting Started with Deep Learning and Neural Network

34

Reminds you of something?

Computational Representation of Perceptron.

​

​

35

Perceptron Learning Algorithm

​

1. Initialize all weights (w) and bias (b) to zero
2. For each training vector x and actual target ‘t’ (desired output)
3. Compute output y from y_in using activation function ‘f’:

​

​

4. Update weights as:
1. Update bias as:
2. Repeat updating weight

and bias until desired output.

Note: α is learning rate, 0 < α ≤ 1

Realizing Gates Using Perceptron

• Perceptron can be trained to show behavior of AND function and OR function because both are linearly separable. But, perceptron can not be trained to learn behavior of XOR function because it is not linearly separable.

Fig: AND Function

Fig: OR Function

Realizing AND gate using Perceptron Algorithm

Example

Train the following perceptron by using given training set

• single layer perceptron can be trained to classify patterns that are linearly separable.

​

• However, XOR function is non-linearly separable as shown in the figure given below. Thus, we can not train single layer perceptron for XOR function. This problem is called XOR Problem.

​

XOR Problem

Minsky and Papert's Analysis (1969).

• The conclusion from Minsky and Papert's analysis:
• Individual elements are weak computational elements.
• Network of elements are required.

Week - 3 - Getting Started with Deep Learning and Neural Network

40

Solving the XOR Problem.

​

• A multi layer perceptron could solve for XOR problem:
• As it can compose arbitrarily complicated Boolean functions!
• In cognitive terms: Can compute arbitrary Boolean functions over sensory input
• We will verify this in Tutorial with some exercise so do not miss the Tutorial.
• More on this in the next week

Week - 3 - Getting Started with Deep Learning and Neural Network

41

Perceptron for Real Value Inputs.

• The step function is unsuitable for gradient-based learning: it is non-differentiable at 0, has zero gradient everywhere else, and gives only a binary output.
• The step function may not capture the subtleties of the data;
• it creates a sharp boundary at 0,
• meaning that all positive sums are classified as class 1
• and all negative sums are classified as class 0.

Week - 3 - Getting Started with Deep Learning and Neural Network

42

The activation function

• In General, Activation Functions are group of function that introduces non-linearity to the output of neurons.
• The activation function does the non-linear transformation to the input, making it capable to learn and perform more complex tasks.

​

​

​

​

​

​

Week - 3 - Getting Started with Deep Learning and Neural Network

43

Artificial Neural Network (ANN)

The – End.

Week - 3 - Getting Started with Deep Learning and Neural Network

45