Training CNNs

Adapted from Deepak Pathak

Slide Credit: Kris Kitani

Slide Credit: Kris Kitani

Slide Credit: Kris Kitani

Slide Credit: Kris Kitani

Slide Credit: Kris Kitani

Slide Credit: Kris Kitani

Slide Credit: Kris Kitani

Slide Credit: Kris Kitani

Last Lecture: CNN Architectures

CNNs: Convolutional Neural Networks

• Layers in Standard CNN
• Convolution
• Pooling
• FC
• Dropout
• Popular Architectures
• LeNet
• AlexNet
• VGG
• (New) GoogleNet
• (New) ResNet

AlexNet: How Many Parameters?

Slide Credit: David Fouhey

AlexNet: How Many Parameters?

Slide Credit: David Fouhey

• 62.4 million parameters
• 6 days to train (2012)
• 8 convolutional layers
• Vast majority are in fully-connected layers
• The paper notes that removing convolutions is disastrous for performance.

AlexNet: Training Trick #1 DropOut

AlexNet: Training Trick #2 Data Augmentation

Slide Credit: Kris Kitani

VGG-16: Going Deeper!

Slide Credit: David Fouhey

nn.Sequential(nn.Conv2d, nn.BatchNorm2d, nn.ReLU, nn.MaxPool2d)

All 3x3 convolutions

VGG-16: What are Convolutions Learning?

What features can you see?

​

VGG-16: Emerging ‘Rule of Thumb’

• Convolutions all standardized to 3x3
• Convolutions always followed by ReLU
• Stack several convolutions at same resolution
• Downsample by 2, increase channels by 2 to create a bottleneck

Receptive Field of 3 Stacks of 3x3 Convolutions?

Slide Credit: Kris Kitani

Receptive Field of 3 Stacks of 3x3 Convolutions?

Slide Credit: Kris Kitani

What do you Gain by Stacking Convolutions?

Slide Credit: Kris Kitani

VGGNet: Training Trick #3 Pre-Training

• Training VGGNet-{13, 16, 19} was incredibly challenging. Why?

​

​

Training Deeper Networks

Every backpropagation step multiplies the gradient by the local gradient

1 * d * d * d … * d = d^n-1

If d << 1, n is big → Vanishing Gradients

Training Deeper Networks

Every backpropagation step multiplies the gradient by the local gradient

1 * d * d * d … * d = d^n-1

If d >> 1, n is big → Exploding Gradients

VGGNet: Training Trick #3 Pre-Training

• Training VGGNet-{13, 16, 19} was incredibly challenging. Why?
• Back-propagation in large networks is unstable

​

• Soln: Train VGG-11

Use weights to initialize VGG-13

Train VGG-13

…

Train VGG-19

​

​

VGGNet: Training Trick #4 Batch Normalization

Slide Credit: Justin Johnson

VGGNet: Training Trick #4 Batch Normalization

Slide Credit: Justin Johnson

Problem #1: What if zero-mean and unit variance is too hard of a constrain?

Slide Credit: Justin Johnson

Slide Credit: Justin Johnson

Problem #2: Mean and variance estimates depend on the mini-batch. We only see one example at a time during test-time.

Batch Normalization at Test Time

Slide Credit: Justin Johnson

Batch Norm at Training V. Test Time

Slide Credit: Justin Johnson

Training

• Take a batch, compute mean and variance
• Normalize input with computed mean and variance
• Feed normalized input to activation layer

Testing

• Compute mean and variance

​

• Normalize input with computed mean and variance
• Feed normalized input to activation layer

Batch Normalization Improves Training

Slide Credit: Justin Johnson

1. Deep networks are much easier to train
2. Allows for higher learning rates, faster convergence
3. Acts as regularization during training
4. Not well-understood theoretically (yet)
5. Behaves differently during training and testing (common bugs)

Alternate Normalization Techniques

GoogLeNet: Why Only 3x3 Convolution?

GoogLeNet: Everything But the Kitchen Sink

Intermediate Loss Layer Prevents Vanishing Gradients

Stagewise training in 3 parts

ImageNet 14’: VGGNet: 7.30% Error, GoogLeNet: 6.67% Error

ResNet: Scaling to 100+ Layers

• Simply stacking 3x3 convolutions (like VGG-Net) does not provide improvement
• 56-layer net has higher training error and test error than 20-layer net

Deep Residual Learning

If x is optimal, we can set weight layers to 0. It's easy to “ignore” extra model capacity.

​

If x is close to optimal, it's easier to find small weights.

​

Ensembling Effect: This could be seen as an ensemble of networks of varying depth

​

Identity serves as a “gradient highway” to preserve strong signal from output to earlier layers during back propagation.

DenseNet: Connecting All Layers with Skip Connections

Training Neural Networks

• Data Normalization
• Weight Initialization
• Activation Functions
• Interpreting Loss Curves
• Adaptive Optimizers
• Regularization
• Feature Normalization
• Model Selection
• Hyperparameter Tuning

​

Input Data Preprocessing: Subtract Mean, Divide by STD

Weight Initialization: Constant Values

Q: What if we initialize all weights/biases to a constant value?

​

Weight Initialization: Constant Values

Q: What if we initialize all weights/biases to a constant value?

A: We cannot break symmetry of weight values. Gradients are the same at each location.

Weight Initialization: Random Values

Q: What if we initialize all weights/biases to a (small) random value?

​

Weight Initialization: Random Values

Q: What if we initialize all weights/biases to a (small) random value?

A: This is usually fine for shallow networks, but becomes problematic for deep networks.

Weight Initialization for Deeper Models

Slide Credit: Kris Kitani

Slide Credit: Rawal Khirodkar

Understanding Loss Curves

Choosing the Correct Learning Rate

Slide Credit: Rawal Khirodkar

Learning Rate Scheduling

• Dropping learning rate helps settle into a local minima
• Increasing learning rate helps get out of a local minima

Learning Rate Scheduling Varieties

• Personal Favorite: Decay on (Validation Accuracy) Plateau

Diagnosing Loss Curves

Diagnosing Loss Curves

Diagnosing Loss Curves

Diagnosing Loss Curves

Diagnosing Loss Curves

Diagnosing Loss Curves

Diagnosing Loss Curves

Diagnosing Loss Curves

Diagnosing Loss Curves

Diagnosing Loss Curves

Diagnosing Loss Curves

Diagnosing Loss Curves

Choosing Hyperparameters

Learning Rate: Look at Loss Curves (1e-3 Safe Bet)

Batch Size: As Large as Possible

# of Epoch: Until Validation Accuracy Saturates

Network Architecture: ResNet (Safe Bet)

Optimizer: Adam (Safe Bet)

For Everything Else: Experimental Validation

• Grid Search
• Random Search
• Bayesian Optimization

Which is better? Why?

Choosing Hyperparameters

Learning Rate: Look at Loss Curves (1e-3 Safe Bet)

Batch Size: As Large as Possible

# of Epoch: Until Validation Accuracy Saturates

Network Architecture: ResNet (Safe Bet)

Optimizer: Adam (Safe Bet)

For Everything Else: Experimental Validation

• Grid Search
• Random Search
• Bayesian Optimization

For low dimensional search spaces, random search works better in practice.

• RS: Search 9 distinct values of the important parameter
• GS: Search 3 distinct values of the important parameter

​

Optimizers: Navigating the Loss Ravines

Drawbacks of Gradient Descent

Slide Credit: Rawal Khirodkar

• Long, narrow ravines

​

• Lots of sloshing around the walls

​

• Only a small derivative in the direction of steepest descent

Regularization: Limit the Complexity of your Model

Regularization: Toy Example

Regularization: Toy Example

​

Regularization: Not Too Little, Not Too Much

​

Training Neural Networks Summary

• Data Normalization
• Weight Initialization
• Activation Functions
• Interpreting Loss Curves
• Adaptive Optimizers
• Regularization
• Feature Normalization
• Model Selection
• Hyperparameter Tuning

Training Neural Networks is an Art and a Science

Personal Experience Debugging Neural Networks

1. Make small delta’s between experiments to localize errors
2. Establish the invariants. What must be true? (Assert Statement)
3. Know what the output should look like. Is the result reasonable?
4. Most bugs stem from the data loader and pre-processing
5. Visualize the output! Get friendly with the pixels
6. Don’t try to reimplement papers. Use off the shelf code whenever possible.
7. Fix the (PyTorch, NumPy, OS) random seed when debugging
8. Log everything! Tensorboard and W&B are great
9. Make sure you are running inference in eval mode
10. Learn to read and google error messages

Complex Error Logs = Simple Bugs