Fundamentals of Deep Neural Networks

1

First Things to Note

• In Machine Learning / Deep Learning

​

• We almost always deal with Mathematical function

​

​

• A model almost always refers to a mathematical function

2

INPUT

F()

Output / Target

ML: Design Pattern

3

Problem statement:

• Know your data well
• What is the input?
• What is your output /target

Define your model:

• Choose a reasonable math function
• Contains unknown parameters

Define the objective function:

• Error / loss / profit

Optimize the objective:

• Use training data and an optimization method to estimate parameters

Now you have all for your model:

• Freeze the model and start using it

Let us Start with a Problem

4

Movie Recommendation

5

• You have two friends John and Mary. You have prior data telling the movies you liked or disliked that they rated. Now a new movie named ‘gravity’ has been released that they rated and you want to decide whether you should watch it.

Know Your Data

6

Sample/example

Features /

input variables

Output / target

Decision Boundary

7

Linear Decision Boundary

8

Class 0

Class 1

Non-linear boundary

9

Optimization: Gradient Descent

10

Review of Differential Calculus

11

What is the sign of here?

What is the sign of here?

​

Local and Global minima

12

x

F(x)

Gradient Descent

13

Training data

Gradient Descent

• Role of gradient descent
• To find the values of parameters that minimize an objective function

14

we want to reach here

Gradient Descent on Multivariate Function

15

Gradient Descent Algorithm

​

​

16

• How to check convergence?
• Two widely used approaches
1. Iterate a fixed number of times
2. Stop if parameters do not change much in two consecutive iteration

​

Limitations of Gradient Descent

17

18

Small value of r

slow convergence

Large value of r

Oscillation / overshooting

Adaptive Gradient Descent

19

Adagrad

• Main Idea:

​

• Learning rate is different for different parameters

​

​

• Learning rate of a parameter depends on its previous gradients

20

Adagrad

21

Adagrad

22

Adaptive Moment Estimation�(ADAM)

23

Deep Neural Network

24

Motivation

25

A Motivating Problem

26

• Movie recommendation:
• You have two friends John and Mary. You have prior data telling the movies you liked or disliked that they rated. Now a new movie named ‘gravity’ has been released that they rated and you want to decide whether you should watch it.

Movie Recommendation

27

• Plot of the data:

​

You liked

You disliked

New movie

Observation: the classes can be separated by straight line

Movie Recommendation

28

• How can you predict the label for new movie, gravity?

​

• Answer: Logistic regression

Logistic Regression: Graphical View

29

f(z)

output

Non-Linear Decision Boundary

Limitations of Linear Decision Function

• Now consider a data like this:

​

​

30

The decision function involves two straight lines

Thus, we need to combine information from two straight lines

Addressing Non-linearity

31

Here is a good network

f

f

f

output

1. The shaded nodes are called hidden nodes / neurons

​

• Each hidden node corresponds to one straight line in the data

Moral of the story:

Hidden nodes help make decision in complex cases

In ML, the complex case means highly non-linear decision boundary that separates the classes

More hidden layers you take, more complex decision your model can make

Structure of Deep Network

32

Structure of Deep Network

33

output

1. One input layer with a set of nodes that take the feature values for a sample
• # nodes in this layer = dimension of the data = # features

​

• One output layer with a set of nodes, that gives the class labels
• # nodes in this layer = # class label in the training data

​

• Multiple hidden layers, each layer contains a set of nodes
• Higher the number of hidden layers, deeper the network

Components of Deep network

Ingredients of a Deep Network

34

Ingredients of a Deep Network

35

Things you need to have in order to describe a deep network:

Ingredients of a Deep Network

36

What does a non-input neuron of the network do?

Each node takes inputs from nodes from previous layer,

linearly combines them,

and then passes through the activation function

Activation Function

37

Deep vs. Shallow Network

38

What is the # features of the data the deep network trying to model?

​

How many class labels are there in that data?

​

How many parameters the deep net has?

Training Deep Network

39

Backpropagation Algorithm���The Pillar of Deep Learning

40

Training Neural Network

• Let us start with the simplest network:
• One feature, 2 class, 2 hidden nodes

​

41

x

1

2

3

Training Neural Network

42

x

1

2

3

Training Neural Network

43

x

1

2

3

• How can we find good values of parameters?
• Answer: By gradient descent

Training Neural Network

44

x

1

2

3

Let us compute gradients:

The chain rule of derivative

Training Neural Network: Computing Gradients

45

x

1

2

3

gradients: the common parts

This method is known as backpropagation

46

1. Initialize the network parameters (weights and biases)

​

• Choose and optimization method (vanilla SGD, ADAM etc)

​

• Repeat the following steps (until you are happy with the result)

​

• Take a forward pass for an input sample

​

• Compute the cost function

​

• Compute the gradients of the cost with respect to parameters using backpropagation

​

• Update each parameter using the gradients, according to the optimization algorithm

​

47

Other Important Details

Parameter Initialization

• Very large initialization leads to exploding gradients

​

• Very small initialization leads to vanishing gradients

​

• We need to maintain a balance

48

Initialization

• Xavier initialization

​

49

Initialization

• Kaiming Initialization

​

50

Computing Loss

51

Cross Entropy

52

Regularization

53

Improving Single Model Performance

54

Regularization:

55

• Regularization techniques are essential in deep learning to prevent overfitting, improve generalization, and ensure that the model performs well on unseen data.
• Overfitting occurs when a model learns the noise in the training data instead of the underlying pattern, leading to poor performance on new, unseen data.

Common Regularization Technique:

• L1 and L2 Regularization (Weight Decay)
• Dropout
• Early Stopping
• Batch Normalization
• Data Augmentation

• Key idea
• Add a term to the error/loss function

56

Regularization: L1 and L2 Regularization (Weight Decay)

• Key idea
• Add a term to the error/loss function

57

Regularization: L1 and L2 Regularization (Weight Decay)

Regularization: Dropout

58

Regularization: Dropout

59

Key Idea: During training, randomly drop some neurons. Probability of dropping is a hyper-parameter.

Srivastava et. al.

Regularization: Early Stopping

60

Key Idea:

​

​

Early stopping involves monitoring the model's performance on a validation set during training and stopping training when the performance stops improving, which prevents overfitting by not allowing the model to train too long on the training data.

Regularization: Batch Normalization

61

Key Idea: Normalizes the inputs of each layer to have zero mean and unit variance.

​

It helps stabilize and accelerate training by reducing internal covariate shift.

Regularization: Data Augmentation

62

Source: Fei Fei Li

Data Augmentation: Image Transformation

63

Source: Fei Fei Li

Data Augmentation: Random Crops and Scales

64

Source: Fei Fei Li

• During training add random crops

​

• Resize training images

​

• Sample random patch

Data Augmentation: Color Changes

65

Source: Fei Fei Li

• Randomize contrast and brightness

Thank you!

66