Artificial Intelligence and Machine Learning

Week 6 - Practical Aspects of Training Deep Neural Network

1

Practical Aspects of Training Deep Neural Network

2

Before Training

Three way hold out Method

3

Train Set: Used for updating weights through backpropagation.

​

Validation Set: Used to monitor the model during training and detect overfitting.

​

Test Set: Used only after training to evaluate generalization performance.

Bias vs Variance

Height

Weight

4

Let's say we have a bunch of weight data and the corresponding heights

Bias vs Variance

Height

Weight

5

Ideally, we’d like to create a mathematical model that describes the relationship between weight and height

Bias vs Variance

Height

Weight

6

First, let’s divide our data into training and testing data

Bias vs Variance

Height

Weight

7

Here are just the training sets

Bias vs Variance

Height

Weight

8

Firstly, lets use linear regression

Bias vs Variance

Height

Weight

9

But here’s the problem. No matter how we try to fit the line, our linear model never shows us the “true” relationship as it won’t curve.

Bias vs Variance

Height

Weight

10

But here’s the problem. No matter how we try to fit the line, our linear model never shows us the “true” relationship as it won’t curve.

Bias vs Variance

Height

Weight

11

But here’s the problem. No matter how we try to fit the line, our linear model never shows us the “true” relationship as it won’t curve.

Bias vs Variance

Height

Weight

12

Thus, the straight line will never show us the “true” relationship no matter how well we try to fit.

Bias vs Variance

Height

Weight

13

This inability for machine learning model to capture the true relationship is called bias. Our straight line model has high bias.

Bias vs Variance

Height

Weight

14

Let’s see another example. This time we fit a curvy line.

Bias vs Variance

Height

Weight

Height

Weight

15

So, when we take training set, clearly our curvy line has zero error and the linear model has high error.

Bias vs Variance

Height

Weight

16

But don’t forget we have test data too. And when you take only test data into account…

Bias vs Variance

Height

Weight

17

Linear regression model is showing a reasonable inaccuracy

Bias vs Variance

Height

Weight

18

But our curvy line is underperforming.

Bias vs Variance

Height

Weight

Height

Weight

19

So, when we take test set, clearly our curvy line has high error and the linear model has reasonable error like in the case of training.

Bias vs Variance

Height

Weight

Height

Weight

20

So, even though our curvy line did good job while training, the error was too high for test set.

Bias vs Variance

Height

Weight

Height

Weight

21

This difference in fits between test and training set for our model is called variance.

Bias vs Variance

Height

Weight

Height

Weight

22

And since our curvy line fits training data well and not testing data, this also means our model is overfitting.

Bias vs Variance

Height

Weight

23

In contrast, since our linear model cannot capture the true relationship even on the training data, this situation is called underfitting.

Bias vs Variance

Height

Weight

Height

Weight

24

The goal is to find a model that has…

Low variance

And low bias

Bias Variance Tradeoff

• Models with high bias tend to have low variance.

• Models with high variance tend to have low bias.

The idea is to find a “sweet spot”.

25

Bias

Variance

Underfitting zone

Overfitting zone

Generalization error

Model complexity

Practical workflow of Deep Learning

26

AI generated image

Dealing With the Overfitting

27

Training With the more Data

• Include more training data
• Simply introducing more data in the training set can reduce overfitting of a complicated model as it previously may not have sufficient data to generalize well

​

​

​

​

But Where to get more data?

28

Data Augmentation

Technique used in deep learning, especially in computer vision to artificially expand a dataset by creating modified versions of images!!

​

​

​

29

Copyright of the images used belongs to creator themselves.

Normalization

30

x

y

x

y

x

y

Mean subtraction

Normalize variance

Copyright of the images used belongs to creator themselves.

Normalization

31

https://ai.stackexchange.com/questions/6860/why-feature-scaling-for-skewed-contour

Copyright of the images used belongs to creator themselves.

Why?

Normalization

Why do we normalize the data in the first place?

y

32

Copyright of the images used belongs to creator themselves.

Batch Normalization

z^[1]

z^[2]

z^[3]

33

Can we normalize the values of z^[1] in order to train w^[1],b^[1] faster?

Given some values of Neural net z^[1], z^[2],...z^[n]�

• Compute mean (μ)�
• Compute variance (σ²)�
• z^[i]_norm = (z[i] - μ)/sqrt(σ²+ε)

w^[1],b^[1]

Copyright of the images used belongs to creator themselves.

Batch Normalization

34

Early Stopping

iterations

error

train error

test error

Stop here

35

Regularization

Regularization refers to techniques that are used to calibrate machine learning models in order to minimize the adjusted loss function and prevent overfitting.

Regularization

36

Lasso Regularization (L1 Regularization)

Like Ridge, it adds a penalty to the loss (Loss function + λΣ|θ|), and it can shrink some weights exactly to zero

Without regularization (overfit)

With regularization (good fit)

37

Ridge Regularization (L2 Regularization)

Loss function + Regularized Term

Mathematically, we introduce a regularization parameter; which penalizes the size of the weights (coefficients).

Height

Weight

λ = 0 => No impact on the model

λ = minimal => acceptable model accuracy

λ = high => very high impact/underfitting

λ is chosen by cross-validation and hyperparameter tuning.

38

Ridge Regularization (L2 Regularization)

In other words, by starting with slightly worse fit, Ridge Regularization can provide better long term predictions.

By increasing bias slightly, the variance can drop significantly.

Height

Weight

39

Dropout

40

Copyright of the images used belongs to creator themselves.

While Training:

Apply a DropOut layer with p = 0.5

Vanishing and Exploding Gradients

41

Early-layer weights barely change.

The model learns very slowly.

Training stagnates at a very early stage just after a few iterations.

Copyright of the images used belongs to creator themselves.

Remember?

Vanishing and Exploding Gradients

​

42

The model weights may become NaN during training.

There is an exponential growth in the model parameter values.

Gradient descent diverges

Copyright of the images used belongs to creator themselves.

Choose Activation Function Wisely

• Use ReLU and its variants as activation function

43

Weight initialization techniques

https://cs230.stanford.edu/section/4/

44

x₁

x₂

x₃

x_n

z

w₁

w₂

w₃

w_n

z = x₁w₁ + x₂w₂ + … + x_nw_n

In order to regulate z (avoid vanishing or exploding), the larger n is, the smaller you want w_i to be

Avoid zero initialization → all neurons get identical updates (symmetry).

Use random initialization → breaks symmetry, so neurons learn different features.

​

Issue with random initialization → Can cause vanishing/exploding

var(w_i) = 1/n

For layer L

W_[L] = np.random.randn(shape)*np.sqrt(1/n_[L-1])

Weight initialization techniques

https://cs230.stanford.edu/section/4/

45

x₁

x₂

x₃

x_n

z

w₁

w₂

w₃

w_n

Some Popular Techniques

Xavier Initialization

Works well for sigmoid and TanH

Weight initialization techniques

https://cs230.stanford.edu/section/4/

46

x₁

x₂

x₃

x_n

z

w₁

w₂

w₃

w_n

Some Popular Techniques

He Initialization for ReLU and its Variants

Mini batch gradient descent

​

for t = 1 … 5000{

forward prop on X^{t}

compute cost J^{t}

backprop to compute gradient w.r.t J^{t}

update w and b

}

1 Epoch

Mini batch gradient descent

X = [ x^[1] x^[2] … x^[1000] x^[1001] … x^[m] ]

Here, the shape of X = (n_x,m)

Y = [ y^[1] y^[2] … y^[1000] y^[1001] … y^[m] ]

Here, the shape of Y = (1,m)

If you have 5 million data… let's create mini-batches of 1000 training examples

What is the shape of one block of X and Y?

…

…

x{1}

x{2}

x{3}

x{t}

y{1}

y{2}

y{3}

y{t}

Momentum

Copyright of the images used belongs to creator themselves.

RMSprop

2

2

W = W - ɑ dW/sqrt(v_dw) , b = b - ɑ db/sqrt(v_db)

Adam

momentum

rms prop

Learning Rate Decay

The idea is to slowly decay the learning rate so that convergence occur.

lr = initial_lr / (1 + decay_rate × epoch_num)

Copyright of the images used belongs to creator themselves.

Hyperparameter Tuning

𝝰

𝛃₁, 𝛃₂, 𝝴 (0.9, 0.999, 10^-8 respectively)

𝛃 (0.9 appx)

Number of layers

Number of hidden units

Learning rate decay

Mini-batch size

Some Popular CNN Architectures

54

Recap: CNN Architecture

55

An Architecture

56

A ConvNet architecture is in the simplest case a list of Layers

that transform the image volume into an output volume (holding the class scores).

Imagenet Dataset

57

contains over 14 million labelled images across ~20,000 categories (the ILSVRC subset uses 1,000 classes)

​

About ImageNet Competition

• The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) was a highly influential annual competition in computer vision, running from 2010 to 2017.

​

• It served as a benchmark for evaluating the performance of image classification and object detection models on the ImageNet dataset

​

• Impact of ImageNet Competition:
• Accelerated deep learning research: Led to CNN dominance in vision tasks.
• Improved AI applications: Helped power self-driving cars, medical imaging, facial recognition, and more.
• Drove industry adoption: Companies like Google, Facebook, and Nvidia heavily invested in AI.
• It ended in 2017, by which time the top-5 error (~2.3%) had dropped below the estimated human-level error (~5%).

​

​

58

ImageNet Major Breakthrough.

59

AlexNet.

• AlexNet is a deep convolutional neural network (CNN) architecture that won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012.
• It was developed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton and marked a breakthrough in deep learning for computer vision.
• Key Features:
• 8 Layers Deep: 5 Convolutional Layers + 3 Fully Connected Layers.
• ReLU activation: Used instead of sigmoid to speed up training and mitigate the vanishing gradient problem.
• DropOut Regularization: Applied in fully connected layers to prevent overfitting.
• Overlapping Max – Pooling: Helps reduce spatial dimensions while maintaining important features.
• Data Augmentation: Techniques like random cropping, flipping and color jittering were used to improve generalization.
• Trained with GPU Acceleration: AlexNet was one of the first deep networks trained on two NVIDIA GPUs to handle large – scale computations efficiently.

​

60

AlexNet Architecture Breakdown.

61

AlexNet Parameters.

62

VGG 16

63

VGG 16

64

VGG – 16 Architecture.

65

VGG – 16 Parameters.

66

GoogLeNet – Inception V1.

67

GoogLeNet – Inception V1.

• Key Innovations in Inception v1 (Contd.):
• Deeper Network with Fewer Parameters:
• 22 layers deep compared to AlexNet (8 layers) and VGG – 16 (16 layers).
• Used only about 7 million parameters, compared to AlexNet's ~60 million.
• Global Average Pooling Instead of Fully Connected Layers:
• Replaced traditional fully connected layers with average pooling, reducing parameters and overfitting.
• Auxiliary Classifiers (Extra Loss Branches):
• Added two smaller classifier branches at intermediate layers to help gradient flow, reducing the vanishing gradient problem in deep networks.

​

68

GoogLeNet – Inception V1 - Architecture.

69

ResNet (Residual Networks)

• ResNet (Kaiming He et. al 2015 winner.) introduces the concept of residual connections or skip connections, which help mitigate the vanishing gradient problem and enable training of deeper networks.
• Residual or Skip Connections:
• Layer inputs are propagated and added to the layer output.
• Mitigate the problem of vanishing gradients during training
• Allow training very deep NN (with over 1,000 layers)
• Several ResNet variants exist: 18, 34, 50, 101, 152, and 200 layers
• Are used as base models of other state-of-the-art NNs.

70

Workings of ResNet.

• Feature Extraction:
• Convolutional Layers:
• Like VGG16, ResNet uses convolutional layers to extract features from the input image, but it also includes residual blocks.
• Residual Block:
• A residual block consists of two or more convolutional layers,
• with a shortcut or skip connection that bypasses these layers and adds the input to the output.
• This helps the model learn the residual mapping
• the difference between the input and the output; instead of the direct mapping, which aids in training very deep networks.
• ResNet uses batch normalization after each convolution to stabilize learning and speed up convergence.

71

Workings of ResNet.

• Feature Extraction:
• Global Average Pooling (GAP):
• Instead of using fully connected layers like VGG16, ResNet uses Global Average Pooling (GAP) to reduce the feature maps from the convolutional layers.
• GAP averages each feature map into a single scalar, drastically reducing the number of parameters and preventing overfitting.
• Classification:
• Softmax Layer:
• The output from GAP is a 1D vector, which is passed through a fully connected layer and then softmax.
• The softmax function converts the class scores into a probability distribution over the possible classes.
• The class with the highest probability is selected as the predicted label.

​

72

ResNet Architecture.

73

ResNet – 50 Parameters.

74

Better Result in Practice.

Transfer Learning

75

Transfer Learning.

• Improvement of learning in a new task through the transfer of knowledge from a related task that has already been learned {Trained}.
• Cheaper, faster way of adapting a neural network by exploiting their generalization properties
• Training your own model can be difficult with limited data and other resources
• It is a laborious task to manually annotate your own training dataset
• Why not reuse already pre-trained models?

​

76

Transfer Learning-Process.

• Start with pre-trained base network.
• Base model is initiated using popular architecture such as ResNet/VGG etc.
• Generally Trained on Imagenet Dataset.
• Downloads the pre-trained weights.

77

Transfer Learning – Process.

• Redesign a decision layer of base model, i.e. add a final layer that is compatible with problem in hand.
• New dataset with C classes

​

78

Transfer Learning – Process.

• Redesign a decision layer of base model, i.e. add a final layer that is compatible with problem in hand.
• New dataset with C classes
• Training of the Model:
• Two options:
• Options 1: If the dataset is Small:
• Freeze most of the base network and {Fine Tune} only the top few layers.
• Since the dataset is small, unfreezing only a few layers reduces the risk of overfitting.
• Use a Small Learning Rate:
• This is crucial!
• A small learning rate (e.g., 1e-4 or 1e-5) prevents drastic weight updates and helps retain useful features learned from the original dataset.

79

Freezing and Fine Tuning.

• Frozen layers do not change weights during training.
• Fine-tuning is done by unfreezing the base model or part of it.
• Re-train the entire model again on the whole dataset at a very low learning rate.
• The low learning rate will increase the performance of the model on the new dataset while preventing overfitting.
• Which layer to re-train?
• Depends on the domain
• Start by re-training the last layers (last fully-connected and last convolutional)
• work backwards if performance is not satisfactory

​

80

Where to find Pre-trained models?

• Keras pre-trained models:
• Classify with base classes (imagenet class) with ResNet.

​

​

​

​

81

Why it Works?

• Intuitively, deep networks learn multiple levels of abstraction i.e. Different layer of convolution learns/extract different features
• Generally low level to high level features
• Low level features such as lines, spot and edges will remain the same for all kind of images

​

82

Closing Remarks.

• Setting up a problem:
• Obtain training data
• use appropriate representations for inputs and outputs.
• Choose network architecture
• More neurons need more data
• Deep is better, but training may be unstable
• Choose appropriate loss function
• Choose regularization and other heuristics like batch norm, dropout etc.
• Choose optimization algorithm.
• E.g. Adam {Discuss this in Workshop}
• Read a lot.
• Happy Training!!!!!

83

Thank You!

84