CS6886: Systems for Deep Learning�Introduction to Deep Learning

Prof. Gopalakrishnan Srinivasan

CSE | BrainSeek Lab | RISE Lab | IITM

​

https://brainseek-lab.github.io/brainseek-lab/index.html

sgopal@cse.iitm.ac.in

CS6886

2

2011

2016

2022+

IBM Watson defeated human champions in Jeopardy

Google AlphaGo defeated Lee Sedol 4-1 in Go

Deep learning

Statistical machine learning algorithms

Evolution of Machine Learning

Apps built using LLMs (ChatGPT, Perplexity, Gemini) can generate text, images, videos, and solve Olympiad problems …

• Deep learning wave enabled by high computational power provided by GPUs and custom accelerators (Google TPU, Tesla Dojo, etc.)
• Nvidia championed the use of GPUs for neural network training after the now famous AlexNet paper

CS6886

3

Deep Learning Revolution Enabled by Hardware

Image reference: Kendall, J.D. and Kumar, S., 2020. The building blocks of a brain-inspired computer. Applied Physics Reviews, 7(1), p.011305.

AlexNet: Krizhevsky, A., Sutskever, I. and Hinton, G.E., 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25.

CS6886

4

General Principle

• Learn when (and how) to use LLM based apps
• Use them as tools to aid in learning and increase productivity rather than drop-in replacement for our brain

Image generated using Perplexity Pro

CS6886

5

Deep Learning Fundamentals

CS6886

6

Fundamental Building Block: Neuron

• Biological neurons produce electrical impulses or spikes when “sufficiently” excited
• Artificial neuron integrates weighted inputs and applies a non-linear activation
• ReLU, sigmoid, tanh, etc.

CS6886

7

Common Activation Functions

• Choice of activation function depends on computational efficiency and “stable” gradients during training

Image reference: https://www.researchgate.net/figure/Commonly-used-activation-functions-a-Sigmoid-b-Tanh-c-ReLU-and-d-LReLU_fig3_335845675

CS6886

8

Feed-Forward Neural Network

• Layers of artificial neurons interconnected by synaptic weights
• Hidden layers learn high-level input representations
• Can we use linear activation in the hidden layers?
• Output layer contain prediction probabilities for classification application

• Human brain learns using “local” learning rules in an unsupervised manner

CS6886

9

• Local learning
• Synaptic weight update depends only on the parameters “locally stored” in the interconnecting neurons
• Weights can be updated during forward propagation
• Reduces computation and communication complexity compared to backpropagation-based global optimization

• Unsupervised learning is the ability to learn input features without using any explicit targets or labels
• Modern DNNs are commonly trained using supervised backpropagation-based methods that minimize error between actual and target outputs

​

Training Algorithm for Biological Neural Nets

Training Algorithm for Deep Neural Networks: Backpropagation

CS6886

10

• Compute loss function at the output layer
• Mean Squared Error (MSE), cross entropy, etc.
• Backpropagate the loss and determine the partial derivative of the loss with respect to the weights
• Update the weights in a direction that minimizes the loss

CS6886

11

Training Algorithm for Deep Neural Networks: Backpropagation

Loss function is denoted by E

CS6886

12

General Principle

• Majority of deep learning operations are vector-matrix, matrix-matrix, or elementwise computations
• Belong to a class of embarrassingly parallel applications

Image generated using Perplexity Pro

CS6886

13

Advances in DNN Architectures

• Implements weight sharing to achieve representation learning using fewer number of parameters
• Redefined state-of-the-art in ImageNet, ushering in deep learning revolution
• Typically consists of convolution, strided convolution, and pooling layers

CS6886

14

LeNet: LeCun, Y., Bottou, L., Bengio, Y. and Haffner, P., 2002. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), pp.2278-2324.

AlexNet: Krizhevsky, A., Sutskever, I. and Hinton, G.E., 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25.

Convolutional Neural Network Architecture

• How to determine the output dimension of a convolution operation?
• Example: HxWx1 input convolved with KxK kernel and stride ‘s’
• How to determine the number of MAC operations to perform a convolution operation?
• Example: HxWxC_in input convolved with KxKxC_inxC_out kernel to produce HxWxC_out assuming unit stride

CS6886

15

Convolutional Neural Network Fundamentals

• Consists of 16 (3x3 conv and fully-connected) layers
• High computational and memory requirements

CS6886

16

VGG architecture: Simonyan, K. and Zisserman, A., 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

Image reference: https://neurohive.io/en/popular-networks/vgg16/

VGG16 Architecture

• Incorporates identity or skip connections between layers
• Identity connections replaced with 1x1 convolution layer in case of dimension mismatch
• Multiple path for gradient to backpropagate in deeper networks
• Example: ResNet101, ResNet152

CS6886

17

ResNet architecture: He, K., Zhang, X., Ren, S. and Sun, J., 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).

Image reference: https://www.researchgate.net/figure/Original-ResNet-18-Architecture_fig1_336642248

ResNet Architecture

CS6886

18

Advances in Backpropagation-based Training of DNNs

• Sigmoid suffers from vanishing gradient problem
• Leads to training convergence issues

CS6886

19

Which Loss Function to Use for Classification: Sigmoid + MSE Loss ✕

Image reference: https://medium.com/@hatodi0945/a-comparison-between-mse-cross-entropy-and-hinge-loss-4d4fe63cca12

• Amplifies the correct prediction and attenuates the incorrect ones

CS6886

20

Which Loss Function to Use for Classification: Softmax + Cross Entropy Loss ✓

Image reference: https://www.geeksforgeeks.org/deep-learning/the-role-of-softmax-in-neural-networks-detailed-explanation-and-applications/

• Partial derivative of the loss with respect to weighted input sum ensures training convergence

CS6886

21

Which Loss Function to Use for Classification: Softmax + Cross Entropy Loss ✓

Image reference: https://wikidocs.net/235711

CS6886

22

Advances in Backpropagation based Training

• Modern frameworks provide different optimizers for performing backpropagation
• Stochastic Gradient Descent (with momentum, weight decay), Adam, etc.
• Weight initialization
• Dropout-based regularization
• Data augmentation pipeline
• Batch normalization

CS6886

23

Weight Initialization is All you Need!

• Layer-wise pre-training was used in the early days of deep learning to provide a good inductive bias before end-to-end backpropagation
• Until we got techniques* to initialize the weights based on a layer parameters

​

​

​

• “n” represents either fan-in (C_in*K_H*K_W) or fan-out (C_out*K_H*K_W)

He initialization: He, K., Zhang, X., Ren, S. and Sun, J., 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision (pp. 1026-1034).

Xavier initialization: Glorot, X. and Bengio, Y., 2010, March. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics (pp. 249-256). JMLR Workshop and Conference Proceedings.

CS6886

24

Dropout based Regularization

• Drop certain fraction of neurons during the training phase
• Akin to training an ensemble of thinned sub-networks
• Need to tune the dropout rate hyperparameter
• No neurons are dropped during inference; instead, the output is scaled proportionally
• Demonstrated to prevent overfitting and improve generalization

Dropout regularization: Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R., 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1), pp.1929-1958.

CS6886

25

Data Augmentation Pipeline

• Dataset is divided into training and test sets or training, validation, and tests sets
• Needed to ensure hyperparameters are not tuned for unknown test data
• Data transformation is applied during training to increase diversity and improve generalization
• Example: normalization, random crop, flipping, etc.

CS6886

26

Batch Normalization for Intermediate Layers

• How to normalize the rest of the DNN layers?
• Normalize with respect to mean and variance obtained over the mini-batch
• Scale and shift the normalized value
• For input x = {x₁, x₂, …, x_N}

​

​

​

• Acts as regularizer and precludes the need for dropout

Batch normalization: Ioffe, S. and Szegedy, C., 2015, June. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning (pp. 448-456). pmlr.

Image reference: https://paperswithcode.com/method/batch-normalization

CS6886

27

Learning Resources

• Sample DNN training and evaluation codes have been made available
• https://colab.research.google.com/drive/1a_pdquM3Bj4lHjnfihjelb01dq5xjzfU?usp=sharing
• https://colab.research.google.com/drive/1fZ9QkHAxhpQ0y8UUwSnkCeZjxTnUbzlG?usp=sharing
• Deep learning course by Prof. Mitesh Khapra
• https://www.cse.iitm.ac.in/~miteshk/CS6910.html
• Deep learning textbook by Ian Goodfellow, Yoshua Bengio, Aaron Courville
• https://www.deeplearningbook.org/