Deep Learning

Relation between AI, ML and DL

sites.google.com/view/rajkumars1987

5

24-Jul-26

rajkumars@vit.ac.in

​

​

​

​

​

Why Deep Learning

sites.google.com/view/rajkumars1987

6

24-Jul-26

rajkumars@vit.ac.in

​

​

​

​

​

Why Deep Learning

Deep Learning

• Deep learning imitates the way our brain works i.e. it learns from experiences. It uses the concepts of neural networks to solve complex problems.
• Informally, a Neural Network with several layers of neurons between input and output layers

​

Deep Learning Applications Across Industries

• Self Driving Cars
• News Aggregation and Fraud News Detection
• Natural Language Processing
• Virtual Assistants
• Entertainment
• Visual Recognition
• Fraud Detection
• Healthcare
• Personalisations
• Detecting Developmental Delay in Children

• Colourisation of Black and White images
• Adding sounds to silent movies
• Automatic Machine Translation
• Automatic Handwriting Generation
• Automatic Game Playing
• Language Translations
• Pixel Restoration
• Photo Descriptions
• Demographic and Election Predictions
• Deep Dreaming

Deep Learning Architectures

Courtesy; https://developer.ibm.com/technologies/artificial-intelligence/articles/cc-machine-learning-deep-learning-architectures/

Deep Learning Architectures (2)

Biological inspiration and neural networks

Example Scenario

Summary

In your brain:

• Each sensory input activates a neuron.
• A sound activates an auditory neuron.
• Footprints activate a visual detection neuron.
• Memory of past tiger encounter activates memory neurons.
• Your brain sums all the signals.
• If the total threat score is high → You RUN!
• If the score is low → You stay calm.

​

Neural Network Perspective

​

Each input = one neuron (input node)

The three circles represent input neurons.

Each arrow = connection with a weight

• Weight shows importance of the input.
• In the forest example:
• “Saw a tiger last week” may have a higher weight
• “Hear a sound” may have medium weight
• “Footprints” may have high weight

Summation (Σ) = neuron combining all signals

The Σ symbol represents a neuron body that:

1. Receives signals
2. Multiplies each by its weight
3. Sums them
4. Applies an activation function (decision rule)

Biological Neurons

Relationship b/w Biological vs Artificial NN

Neuron Model

Evolution of Neural Networks

Basic models of ANN

• The models of ANN has three basic entities are

- the models synaptic interconnections

- the training or learning rules for updating the connection weights

- their activation functions

Connections

Notes : ANN connections include fully connected, partially connected, feedforward,

feedback (recurrent), lateral, and self-connections, each defining how neurons

are linked and how information flows.

Learning

Note : “Learning in ANN is the process of updating weights and bias based on error to improve the network’s performance.”

Activation functions

• Activation functions are an integral building block of neural networks that enable them to learn complex patterns in data.
• They transform the input signal of a node in a neural network into an output signal that is then passed on to the next layer.
• Without activation functions, neural networks would be restricted to modeling only linear relationships between inputs and outputs.
• Activation functions introduce non-linearities, allowing neural networks to learn highly complex mappings between inputs and outputs.
• Activation functions enable neural networks to learn these non-linear relationships by introducing non-linear behaviors through activation functions.
• This greatly increases the flexibility and power of neural networks to model complex and nuanced data.

Activation function

Activation function

Activation function

Why Do We Need Activation Functions?

• An activation function determines whether a neuron should be activated and what output it should produce. It introduces nonlinearity into a neural network, enabling it to learn complex patterns.
• Without activation functions, a neural network would behave like a simple linear model, regardless of how many hidden layers it has.

​

Why Are Activation Functions Important?

1. Introduce Nonlinearity

​

• Real-world problems are usually nonlinear.

Examples:

• Face recognition
• Speech recognition
• Handwriting recognition
• Stock price prediction

Activation functions allow neural networks to learn these complex relationships.

​

Why Are Activation Functions Important?

2. Enable Learning of Complex Patterns

​

Each hidden layer extracts more advanced features.

Image -> Edges -> Shapes -> Face

​

Without activation functions, the network cannot learn these hierarchical features.

Why Are Activation Functions Important?

3. Prevent the Network from Behaving Like Linear Regression

​

Suppose a network has several hidden layers:

​

Input → Hidden Layer → Hidden Layer → Output

​

If every neuron uses only a linear function:

y=Wx+b then the entire network is mathematically equivalent to one linear transformation:

Input ─────────────► Output

This means adding more layers provides no extra learning power.

Why Are Activation Functions Important?

4. Control the Output Range

Different activation functions produce outputs in different ranges.

​

​

These ranges make activation functions suitable for different tasks.

Why Are Activation Functions Important?

5. Improve Training Efficiency

Modern activation functions such as ReLU help reduce the vanishing gradient problem, allowing deep networks to train faster and more effectively.

Activation function

Without vs With Activation Function

• Without Activation Function

• With Activation Function

​

​

Example without and with activation in network

• Note : “An activation function is a mathematical function applied to the neuron’s output to introduce non-linearity and determine the final output.”

Loss Functions

• A loss function measures how far a neural network's prediction is from the actual (true) value. It tells the model how much error it has made during training.
• Low loss → Better prediction
• High loss → Poor prediction
• The goal of training is to minimize the loss by updating the network's weights.

​

Loss Functions

Function Approximation in Neural Networks

• Function approximation is one of the fundamental ideas behind neural networks. It means that a neural network learns a mathematical function that maps inputs to outputs.
• Mathematically,
• y=f(x)
• where:
• x = input
• f = function learned by the neural network
• y = predicted output
• The goal of training is to make the learned function f(x)f(x)f(x) as close as possible to the true relationship between inputs and outputs.

​

Sample of Activation usage in Neural Network

Classification and Clustering Problems

Classification and clustering are two important machine learning tasks. Although both involve grouping data, they differ in how they learn from the data.

Classification and Clustering: Examples and Common Algorithms

Simple Illustration

• Classification

​

• Clustering

​

Deep Networks Basics

• A Deep Neural Network (DNN) is a neural network with multiple hidden layers between the input and output layers. These additional hidden layers enable the network to learn increasingly complex patterns from data.
• Simple Neural Network:

-1 hidden layer

• Deep Neural Network:

-2 or more hidden layers

Shallow Neural Networks

• A Shallow Neural Network (SNN) is a neural network that has only one hidden layer between the input layer and the output layer.
• Although it contains only one hidden layer, it can still learn nonlinear relationships using activation functions such as ReLU, Sigmoid, or Tanh.

​

Deep Network vs Shallow Network

Common Activation Functions

Summary

​

Artificial Intelligence isn't magic—it's mathematics, algorithms, and practice. Keep learning, keep experimenting, and you'll be amazed at what you can create.

​

​

​

​

​

Thank you

​

​

"A journey of a thousand miles begins with a single step." — Lao Tzu

Today, you've taken your first step into the world of AI.

• Activation & Loss functions

Loss Vs Error

Activation functions

• Activation functions are an integral building block of neural networks that enable them to learn complex patterns in data.
• They transform the input signal of a node in a neural network into an output signal that is then passed on to the next layer.
• Without activation functions, neural networks would be restricted to modeling only linear relationships between inputs and outputs.
• Activation functions introduce non-linearities, allowing neural networks to learn highly complex mappings between inputs and outputs.
• Activation functions enable neural networks to learn these non-linear relationships by introducing non-linear behaviors through activation functions.
• This greatly increases the flexibility and power of neural networks to model complex and nuanced data.

Types of Activation Functions

Plot Sigmoid Function in the range(-5, 5)

Plot Sigmoid Function in the range(-5, 5)

Why is this a problem?

Because the outputs are always positive:

• The gradients flowing backward tend to have the same sign.
• This can cause zig-zag updates during gradient descent, making optimization slower.
• It also introduces a bias in the activations.

Plot Tanh Function

Plot Tanh Function

Plot ReLu Function

Plot ReLu Function

ReLu Function

• Problems:

​

• 1. Dying ReLU Problem
• 2. Not Zero-Centered
• 3. Unbounded Positive Outputs
• 4. Sensitive to Learning Rate

​

Why is ReLU still preferred?

Despite these issues, ReLU has significant advantages:

• ✅ Very fast to compute
• ✅ Helps avoid the vanishing gradient problem for positive inputs
• ✅ Leads to sparse activations (many outputs become zero)
• ✅ Trains deep networks much faster than sigmoid or tanh

Variants that fix the dying ReLU problem

Softmax Function

• Softmax is used in the output layer for multi-class classification problems.
• Each neuron corresponds to one class and produces a probability score.
• It converts raw neuron outputs (logits) into values between 0 and 1.
• The sum of all output probabilities is always equal to 1.
• Outputs can be directly interpreted as class confidence scores.
• The class with the highest probability is selected as the final prediction.
• Softmax introduces competition among output neurons.
• Increasing the probability of one class automatically reduces others.

Softmax Function

Softmax function

Error Functions

• A lower error indicates that the model's predictions are closer to the actual values, leading to better performance.

​

• Minimizing the error function is crucial because it directly impacts the model's ability to generalize to new, unseen data.

​

• Error functions, or loss functions, measure the difference between the predicted output of a model and the actual target values.

​

• They provide a quantitative measure of how well the model is performing.

​

• The goal during training is to minimize this error, thereby improving the model's accuracy.

Loss Function vs Cost Function

• The loss function calculates the error for a single data point. (sample), the cost function calculates the loss for the entire dataset.

​

• The cost of a neural network is nothing but the sum of losses on individual training samples.

​

• The terms loss and cost are often used interchangeably,

Loss Functions

• Mean Absolute Error (MAE)
• Mean Squared Error (MSE)
• Root Mean Squared Error (RMSE)
• Categorical Cross Entropy
• Binary Cross Entropy
• Sparse Categorical Cross Entropy

Mean Absolute Error (MAE)

• Calculates the average of the absolute differences between predicted and actual values, providing a more robust measure against outliers

Example: MAE

Here

X is input,

W is weight, [Assume w=0.8] Y is the network’s output and t is the target value of the training data.

Example: MAE

MAE=(1.69+4.78+3.93+10.25)/4

=20.65/4=5.1625

Mean Square Error

• Calculates the average of the squares of the errors, giving a higher penalty to larger errors.

Example: MSE

Here

X is input,

W is weight, [Assume w=0.8] Y is the network’s output and t is the target value of the training data.

Example: MSE

• MSE=(2.86+22.8+15.4+105)/4=36.6

Root Mean Square Error (RMSE)

• It measures the average magnitude of the errors between the predicted values and the actual values.

Example: MSE

Here

X is input,

W is weight, [Assume w=0.8] Y is the network’s output and t is the target value of the training data.

Example: RMSE

• MSE=(2.86+22.8+15.4+105)/4=36.6
• RMSE=√36.6=6.05

Binary Cross Entropy Loss

Example: Assume n=1[Only one sample]

Categorical Cross Entropy Loss

• Categorical Encoding is used

Sparse Categorical Crossentropy

• The categorical cross-entropy becomes very memory in-efficient when we have a large number of classes, say 1000.
• This means we have a large array of all zeros and a single 1. In such cases, we use the sparse categorical cross entropy loss function.
• This loss function works on label-encoded data instead of one-hot encoded data, which makes computation very fast when working with a large number of classes.

Sparse Categorical Crossentropy

Perceptron neural networks

What is Perceptron?

A Perceptron is one of the simplest and most fundamental unit of artificial neural networks, introduced by Frank Rosenblatt in 1957.

• The perceptron is a liner model for supervised learning.
• It is used for linear classification.

The perceptron consists of following component's

• Inputs
• Weights
• Bias
• Weighted Sum
• Activation Function
• Output

​

​

​

​

​

Architecture

Perceptron Learning Algorithm

• Step 1 : Initialize all weights and Bias
• Initialize weights w₁, w₂, ..., w_n ≈ small random values
• Initialize bias b ≈ 0
• Choose learning rate η (0 < η ≤ 1)

​

• Step 2 : Input representation
• Input vector: x = (x₁, x₂, ..., x_n)
• True label: y ∈ {0,1} or {-1,1}
• Step 3 : Compute response of the network (Weighted Sum)
• z = w·x + b
• z = Σ_i(w_i x_i) + b

​

Perceptron Learning Algorithm

• Step 4 : Activation Function

​

​

​

• Step 5 : Compute Error
• Error = y - ŷ
• Step 6 : Update Weights and bias
• w_i(new) = w_i(old) + η(y - ŷ)x_i
• b(new) = b(old) + η(y - ŷ)
• Step 7 : Repeat
• Repeat steps 2–6 for all training samples
• Continue for multiple epochs until:
• Error = 0 (perfect classification), OR
• Maximum iterations reached

Example

• Implement the AND function using Perceptron network for binary inputs and targets with 3 epochs

​

​

• Step 1 : Initialize all weights and Bias
• w1​=0,w2​=0,b=0,η=1
• Step 2 : Input representation

​

​

​

​

• Step 3 -6 : Formula

​

Epoch 1

Epoch 2

Epoch 3

Limitation of Perceptron

• Cannot Solve Non-Linear Problems

👉 Biggest limitation

- Perceptron can only draw a straight line (linear boundary)

- Cannot handle problems like XOR

Example:

- AND → ✔ works

- XOR → ❌ fails

• Only Binary Classification

-Output is only:0 or 1

-Cannot directly handle: Multi-class problems, Continuous outputs

​

​

Limitation of Perceptron

• Requires Linearly Separable Data

- Works only if data can be separated by a single line

- If data overlaps: Perceptron will never converge

• No Probability Output

-Gives only hard decision (0 or 1)

-Does NOT provide: Confidence, Probability

• Sensitive to Learning Rate

- If learning rate is:

• Too large → unstable
• Too small → slow learning

​

Limitation of Perceptron

• Limited Learning Capacity

- Single-layer perceptron:

• Cannot learn complex patterns
• No hidden layers
• No Probability Output
• Bias Toward Linear Decision Boundary

- Always assumes:

- Real-world data is often non-linear

​

Thank you

Gradient Descent

Definition

• Gradient Descent is an optimization algorithm used to minimize the loss (error) of a neural network by updating its weights and biases.
• Its goal is to find the best values of the weights that produce the lowest possible error

​

​

​

​

​

Why Do We Need Gradient Descent?

• When a neural network makes a prediction, it may not be correct.
• Example:

Actual House Price: $300,000

Predicted Price: $250,000

Error = $50,000

• Gradient Descent updates the weights so that future predictions become more accurate.

​

​

​

​

​

How Gradient Descent Works

• Step 1: Initialize Weights : Randomly assign initial weights
• Step 2: Forward Propagation : Use the current weights to make a prediction
• Step 3: Calculate Loss : Compare the predicted output with the actual output
• Step 4: Compute Gradient

Calculate how much each weight contributed to the error.

Large gradient → Large weight update

Small gradient → Small weight update

​

• Step 5: Update Weights : The weight update formula is:

​

​

​

• Step 6: Repeat : Repeat the process until the loss becomes very small.

​

​

​

​

Mountain Analogy

• Imagine standing on top of a mountain in heavy fog.
• Your goal is to reach the lowest point (valley).
• The mountain = Loss Function
• The height = Error
• The slope (gradient) tells you which direction is downhill.
• You take small steps downhill until you reach the bottom.

​

​

​

​

• Gradient Descent works in the same way—it follows the slope to reach the minimum error.

How to Compute the Gradient in Neural Networks

• The gradient tells us how much the loss changes when a weight changes. It is the partial derivative of the loss function with respect to a weight.
• Mathematically,

​

​

​

​

How to Compute the Gradient in Neural Networks

• Step 1 : Loss Function

​

• Step 2: Prediction Equation

​

​

How to Compute the Gradient in Neural Networks

• Step 3: Differentiate Step by Step

​

​

​

​

​

How to Compute the Gradient in Neural Networks

• Step 3: Differentiate Step by Step

​

​

​

​

​

How to Compute the Gradient in Neural Networks

• Step 4: Apply the Chain Rule

​

​

​

How to Compute the Gradient in Neural Networks

​

​

​

Types of Gradient Descent

​

​

​

Learning Rate (η)

• The learning rate (η) is a hyperparameter that controls how much the weights are updated during each iteration of Gradient Descent.
• The learning rate determines the size of each step during weight updates.
• Small Learning Rate

Minimum

Slow learning

More Stable

Takes longer to converge

• Large Learning Rate

Maximum

Faster updates

May miss the minimum

Can become unstable

Learning Rate (η)

• Good Learning Rate

Minimum

Faster convergence

Stable learning

• Example

Why Can't We Just Update the Weight by the Gradient?

• Suppose: Weight (w) = 5
• Gradient (∂L/∂w) = 20
• If we update the weight without using a learning rate:

wnew = wold − Gradient = 5 − 20 = −15

• The weight changes from 5 to −15, which is a very large jump.
• Problems:

-The update is too large.

- The model may overshoot (miss) the minimum loss.

• Training can become unstable or even fail to converge.

Therefore, we introduce a Learning Rate (η) to control the size of the update.

• Example with Learning Rate
• Suppose: Weight (w) = 5
• Gradient (∂L/∂w) = 20
• Learning Rate (η) = 0.1
• Gradient Descent Formula:

wnew = wold − η(∂L/∂w)

• Substitute the values:

wnew = 5 − (0.1 × 20) = 5 − 2 = 3

​

The weight changes smoothly from 5 to 3 instead of making a large jump to −15.

Conclusion:

• Without a learning rate → Very large updates → May overshoot the minimum.

• With a learning rate → Controlled updates → Stable and efficient learning.

Multi-Layer Perceptron (MLP)

What is MLP?

• A Multi-Layer Perceptron (MLP) is an advanced neural network made of multiple layers of perceptrons.
• It overcomes the limitation of single perceptron (can solve non-linear problems like XOR)

​

​

​

​

​

Structure of MLP

Layers:

• Input Layer
• Takes input features
• Hidden Layer(s)
• Performs computations
• Can be one or more layers
• Output Layer
• Produces final result

MLP Properties

• Every layer except the input layer includes a bias neuron.
• It is fully connected to the next layer
• No connections within a layer
• No direct connections between input and output layers
• When an ANN has two or more hidden layers, it is called a deep neural network (DNN)

​

Note : 1. Number of output units need not equal number of input units

2. Number of hidden units per layer can be more or less than input or output units

​

114

What do each of the layers do?

1st layer draws linear boundaries

​

​

2nd layer combines the boundaries

​

​

​

3rd layer can generate arbitrarily complex boundaries

Note : 1. Perceptron : Can draw only single line

2. MLP : Combines multiple lines → forms curves, polygons, complex regions

MLP with Back propagation

• Rumelhart  introducing the back propagation training algorithm for MLP.
• Back propagation has two phases: Forward Propagation and Backward Propagation
• Forward Propagation:
• Input → Hidden layer
• Apply weights + bias
• Apply activation
• Pass to next layer
• Final output generated

MLP with Back propagation

• Backward Propagation:

Core learning algorithm of MLP

Step

1. Compute loss
2. Calculate error
3. Propagate error backward
4. Update weights using gradient descent

MLP with Back propagation Algorithm

• Step 1 : Initialize weights and Biases randomly
• Initialize weights w₁₁, w₁₂, ..., w_ij ≈ small random values
• Initialize biases b₁,b₂,…,b_j ≈ small random values
• Choose learning rate η (0 < η ≤ 1)
• Step 2 : Forward Propagation Phase
• For each layer:
• Hidden layer
• Step 2.1 : Each input unit x = (x₁, x₂, ..., x_n) receives input signal and broadcasts to all units in hidden layer
• Step 2.2 : Each hidden unit z = (z₁, z₂, ..., z_j) sums its weighted input signals

z_j = ∑w_ijx_i+b_j

• Step 2.3 Apply activation function

a_j = f(z_j)

​

MLP with Back propagation Algorithm

• Output layer
• Step 2.4 : Each output unit z = (z₁, z₂, ..., z_k) sums its weighted input signals

z_k = ∑w_jka_j+b_k

• Step 2.5 Apply activation function

y_p = f(z_k)

Step 3 : Backward Propagation Phase

• Step 3.1 Compute Loss

Example (Mean Squared Error):

E = ½ * (y-y_p)²

MLP with Back propagation Algorithm

• Step 3.2 : Output layer error

δ_k=(y−y_p)f′(y_p)

• Step 3.3 Hidden layer error

δ_j​=f′(a_j​)∑δ_k ​w_jk​

Step 4 : Update weights

Step 4.1 Output weights:

w_jk(new) ​=w_jk(old)​+ηδ_k​a_j​

Step 4.2 Hidden weights:

w_ij(new​)=w_ij(old)​+ηδ_j​x_i​

Step 4.3 biases

b(new)=b(old)+ηδ

MLP with Back propagation Algorithm

• Step 6: Repeat
• Repeat steps 2–4 for:
• All training samples
• Multiple epochs

​

• Until:
• Error is minimized OR
• Max iterations reached

​

MLP with Back propagation Algorithm

• Algorithm Summary (Simple Steps)
• Initialize weights & bias
• Perform forward propagation
• Compute error
• Backpropagate error
• Update weights & bias
• Repeat until convergence

​

Example

• Network Structure
• Inputs: x₁=1,  x₂=0
• Hidden layer: 2 neurons
• Output: 1 neuron
• Target:

y=1

• Learning rate:

η=0.1

• Activation: Sigmoid

Example

• Step 1: Initialize Weights
• Input → Hidden

w₁=0.2,  w₂=0.3,  b₁=0.1

w₃=0.4,  w₄=0.1,  b₂=0.1

• Hidden → Output

w₅=0.3,  w₆=0.2,  b₃=0.1

• Step 2: Forward Propagation
• Hidden Neuron 1

z₁=(1)(0.2)+(0)(0.3)+0.1=0.3

a₁​=σ(0.3)=0.574

Example

• Hidden Neuron 2

z₂=(1)(0.4)+(0)(0.1)+0.1=0.5

a₂​=σ(0.5)=0.622

• Output Neuron

z₃=(0.574)(0.3)+(0.622)(0.2)+0.1

z₃=0.172+0.124+0.1=0.396

y_p=σ(0.396)=0.598

Example

• Step 3 : Backward Propagation
• Error Calculation

E=1/2​(1−0.598)²=0.081

• Output Layer Gradient (or) Output Layer Error δ₃​=(y−y_p​)⋅y_p​(1−y_p​)

δ₃=(1−0.598)(0.598)(0.402)

δ₃​≈0.097

Example

• Hidden Layer Gradient (or) Hidden Layer Error Error Calculation

Neuron 1:

δ₁​=δ₃​⋅w₅​⋅a₁​(1−a₁​)

δ₁=0.097⋅0.3⋅(0.574)(0.426)

δ₁​≈0.007

Neuron 2:

δ₂​=δ₃​⋅w₆​⋅a₂​(1−a₂​)

δ₂=0.097⋅0.2⋅(0.622)(0.378)

δ₂​≈0.005

Example

• Step 5: Update Weights

Hidden → Output:

w₅​=0.3+0.1(0.097)(0.574)≈0.306 w₆=0.2+0.1(0.097)(0.622)≈0.206

b₃ = 0.1 + 0.1(0.097) ≈ 0.109

Example

• Input → Hidden

Neuron 1 :

w₁​=0.2+0.1(0.007)(1)≈0.201

w₂=0.3+0.1(0.007)(0)=0.3

b₁ =0.1+0.1(0.007)≈ 0.101

Neuron 2:

w₃​=0.4+0.1(0.005)(1)≈0.401

w₄=0.1+0.1(0.005)(0)=0.1

b₂​=0.1+0.1(0.005)≈0.1005

Example : Final Updated Values (After 1 Backpropagation Step)

MLP Algorithm Summary

MLP Algorithm Summary

Thank you

Parameters

• Definition

Parameters are values learned automatically by the neural network during training.

• They determine how the network makes predictions.
• Examples

Weights (W)

Biases (b)

• Characteristics

Learned from data.

Updated during backpropagation.

Directly affect predictions.

​

Hyperparameters

• Definition

- Hyperparameters are values chosen before training.

- They control how the neural network learns, but they are not learned from the data.

​

Summary

• Learned the fundamentals of Neural Networks and Deep Neural Networks.
• Understood the importance of activation functions, loss functions, and function approximation in neural network learning.
• Distinguished between classification and clustering problems and their real-world applications.
• Compared shallow and deep neural networks and studied their architectures.
• Learned how Gradient Descent, Forward Propagation, and Backpropagation train neural networks by minimizing prediction error.
• Understood the difference between parameters (weights and biases) and hyperparameters (learning rate, epochs, batch size, etc.) and their role in model training.

Key Takeaways

• Neural networks learn patterns by adjusting weights and biases.
• Activation functions enable neural networks to learn complex, nonlinear relationships.
• Loss functions quantify prediction errors during training.
• Gradient Descent minimizes the loss by updating model parameters.
• Backpropagation efficiently computes gradients using the chain rule.
• Deep Neural Networks can solve more complex problems than shallow networks.
• Parameters are learned during training, whereas hyperparameters are chosen before training.
• A strong understanding of these fundamentals is essential for studying advanced deep learning models.

Thank you

​

"Deep learning is not about making machines think like humans; it's about teaching machines to learn from data. The stronger your foundation, the greater your innovations."

​

Additional problem

• Network Structure

x1,x2→h1,h2→o

• ​ Assume the following values.
• Inputs

x1=0.5, x2=0.3

• Target t=1
• Weights
• Input → Hidden:
• w1=0.2, w2=0.4
• w3=0.3,w4=0.1
• Hidden → Output:
• w5=0.5,w6=0.6

​

• For simplicity, assume all biases are zero:

b1=b2=b3=0

• Learning rate:

η=0.1

Hidden Layer activation Tanh

Output Layer activation Sigmoid