Summarizing CNNs
Ashis Kumer Biswas, Ph.D.
What are CNNs?
We want to learn the weights of filters to discover features
CNNs are everywhere...
Recommendation Systems
Drug Discovery
Physics simulations
Recommending similar fashion products
CNNs are everywhere...
Recommendation Systems
Drug Discovery
Physics simulations
Izhar Wallach, Michael Dzamba, Abraham Heifets, AtomNet: A Deep Convolutional Neural Network for Bioactivity Prediction in Structure-based Drug Discovery, arxiv 2016
CNNs are everywhere...
Recommendation Systems
Drug Discovery
Physics simulations
Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, Ken Perlin, Accelerating Eulerian Fluid Simulation With Convolutional Networks, arxiv 2016
ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
Task 1: Image classification: Predict the classes of objects present in an image.
Task 2: Single-object localization: Image classification + draw a bounding box around one example of each object present.
Task 3: Object detection: Image classification + draw a bounding box around each object present.
ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
Task 1: Image classification: Predict the classes of objects present in an image.
Task 2: Single-object localization: Image classification + draw a bounding box around one example of each object present.
Task 3: Object detection: Image classification + draw a bounding box around each object present.
We’re focusing on ImageNet
Gives us a common task to compare architectures
Networks trained on ImageNet are often starting points for other vision tasks
Architectures that perform well on ImageNet have been successful in other domains
Alfredo Canziani & Eugenio Culurciello, An Analysis of Deep Neural Network Models for Practical Applications, arXiv 2016
We’re focusing on ImageNet
Gives us a common task to compare architectures
Networks trained on ImageNet are often starting points for other vision tasks
Architectures that perform well on ImageNet have been successful in other domains
Example applications:
We’re focusing on ImageNet
Gives us a common task to compare architectures
Networks trained on ImageNet are often starting points for other vision tasks
Architectures that perform well on ImageNet have been successful in other domains
Overview
The presentation is organized into three stages:
Topics covered:
Overview
The presentation is organized into three stages:
Topics covered:
Overview
The presentation is organized into three stages:
Motivation:
Overview
The presentation is organized into three stages:
Motivation:
Stage 1: Revisiting the Basics
Review: Fully Connected Layers
Takes N inputs, and outputs M units
Each output is a linear combination of inputs
Usually implemented as multiplication by an (N x M) matrix
An example of fully connected layer.
Review: Fully Connected Layers
Takes N inputs, and outputs M units
Each output is a linear combination of inputs
Usually implemented as multiplication by an (N x M) matrix
An example of fully connected layer.
Review: Fully Connected Layers
Takes N inputs, and outputs M units
Each output is a linear combination of inputs
Usually implemented as multiplication by an (N x M) matrix
An example of fully connected layer.
How fully connected layers fix input size
Convolutions can be thought of as sliding fully connected layers
When the inputs to a convolutional layer are larger feature maps, outputs are larger feature maps
Fully connected layers have a fixed number of inputs/outputs, forcing the entire network’s input shape to be fixed
Image source: http://deeplearning.net/software/theano_versions/dev/tutorial/conv_arithmetic.html
Think of this as fully connected layer that takes n x n inputs sliding across the input
How fully connected layers fix input size
Convolutions can be thought of as sliding fully connected layers
When the inputs to a convolutional layer are larger feature maps, outputs are larger feature maps
Fully connected layers have a fixed number of inputs/outputs, forcing the entire network’s input shape to be fixed
Image source: http://cs231n.github.io/convolutional-networks/
The number of output feature maps corresponds to the number of outputs of this fully connected layer
How fully connected layers fix input size
Convolutions can be thought of as sliding fully connected layers
When the inputs to a convolutional layer are larger feature maps, outputs are larger feature maps
Fully connected layers have a fixed number of inputs/outputs, forcing the entire network’s input shape to be fixed
256x256 128x128 32x32 16x16 4x4
What are the spatial resolutions of feature maps if we input a 512 x 512 images?
Image source: http://www.ais.uni-bonn.de/deep_learning/
How fully connected layers fix input size
Convolutions can be thought of as sliding fully connected layers
When the inputs to a convolutional layer are larger feature maps, outputs are larger feature maps
Fully connected layers have a fixed number of inputs/outputs, forcing the entire network’s input shape to be fixed
512x512 256x256 64x64 32x32 8x8
The spatial resolutions are doubled in both dimensions for all feature maps.
Image source: http://www.ais.uni-bonn.de/deep_learning/
How fully connected layers fix input size
Convolutions can be thought of as sliding fully connected layers
When the inputs to a convolutional layer are larger feature maps, outputs are larger feature maps
Fully connected layers have a fixed number of inputs/outputs, forcing the entire network’s input shape to be fixed
512x512 256x256 64x64 32x32 8x8
What happens to to the fully connected layers when input dimensions are doubled?
Image source: http://www.ais.uni-bonn.de/deep_learning/
How fully connected layers fix input size
Convolutions can be thought of as sliding fully connected layers
When the inputs to a convolutional layer are larger feature maps, outputs are larger feature maps
Fully connected layers have a fixed number of inputs/outputs, forcing the entire network’s input shape to be fixed
512x512 256x256 64x64 32x32 8x8
It becomes unclear how the larger feature maps should feed into the fully connected layers.
Image source: http://www.ais.uni-bonn.de/deep_learning/
Fully Connected Layers => Convolutions
Based on the relationships between fully connected layers and convolutions we just discussed, can you think of a way to convert fully connected layers to convolutions?
By replacing fully connected layers with convolutions, we will be able to output heatmaps of class probabilities
Jonathan Long, Evan Shelhamer, Trevor Darrell, Fully Convolutional Networks for Semantic Segmentation, arXiv preprint 2016
Convolutionize VGG-Net
VGG-Net’s final pooling layer outputs 7x7 feature maps
VGG-Net’s first fully connected layer ouputs 4096 units
What should the spatial size of the convolutional kernel be?
How many output feature maps should we have?
Image source: http://www.robots.ox.ac.uk/~vgg/research/very_deep/
Convolutionize VGG-Net
VGG-Net’s final pooling layer outputs 7x7 feature maps
VGG-Net’s first fully connected layer ouputs 4096 units
What should the spatial size of the convolutional kernel be?
How many output feature maps should we have?
Image source: http://www.robots.ox.ac.uk/~vgg/research/very_deep/
Convolutionize VGG-Net
VGG-Net’s final pooling layer outputs 7x7 feature maps
VGG-Net’s first fully connected layer ouputs 4096 units
What should the spatial size of the convolutional kernel be?
How many output feature maps should we have?
Image source: http://www.robots.ox.ac.uk/~vgg/research/very_deep/
Convolutionize VGG-Net
VGG-Net’s final pooling layer outputs 7x7 feature maps
VGG-Net’s first fully connected layer ouputs 4096 units
What should the spatial size of the convolutional kernel be?
How many output feature maps should we have?
Image source: http://www.robots.ox.ac.uk/~vgg/research/very_deep/
Convolutionize VGG-Net
VGG-Net’s final pooling layer outputs 7x7 feature maps
VGG-Net’s first fully connected layer ouputs 4096 units
What should the spatial size of the convolutional kernel be?
How many output feature maps should we have?
Image source: http://www.robots.ox.ac.uk/~vgg/research/very_deep/
Convolutionize VGG-Net
What just happened?
The final pooling layer still outputs 7x7 feature maps
But the first fully connected layer has been replaced by a 7x7 convolution outputting 4096 feature maps
The spatial resolution of these feature maps is 1x1
First and hardest step towards convolutionalization is complete!
Jonathan Long, Evan Shelhamer, Trevor Darrell, Fully Convolutional Networks for Semantic Segmentation, arXiv preprint 2016
Convolutionize VGG-Net
How do we convolutionalize the second fully connected layer?
The input to this layer is 1x1 with 4096 feature maps
What is the spatial resolution of the convolution used?
How many output feature maps should be used?
Jonathan Long, Evan Shelhamer, Trevor Darrell, Fully Convolutional Networks for Semantic Segmentation, arXiv preprint 2016
Convolutionize VGG-Net
How do we convolutionalize the second fully connected layer?
The input to this layer is 1x1 with 4096 feature maps
What is the spatial resolution of the convolution used?
How many output feature maps should be used?
Same idea used for the final fully connected layer
Jonathan Long, Evan Shelhamer, Trevor Darrell, Fully Convolutional Networks for Semantic Segmentation, arXiv preprint 2016
Results
Now, all the fully connected layers have been replaced with convolutions
When larger inputs are fed into the network, network outputs grid of values
The grid can be interpreted as class conditional heatmaps
Jonathan Long, Evan Shelhamer, Trevor Darrell, Fully Convolutional Networks for Semantic Segmentation, arXiv preprint 2016
Global Average Pooling
Take the average of each feature map and feed the resulting vector directly into the softmax layer
Advantages:
Min Lin, Qiang Chen, Shuicheng Yan
Network In Network
Global Average Pooling
In practice, the global average pooling outputs aren’t sent directly to softmax
It’s more common to send the filter wise averages to a fully connected layer before softmax
Used in some top performing architectures including ResNets and InceptionNets
Min Lin, Qiang Chen, Shuicheng Yan Network In Network
Rescaling Demo:
I fed this picture of an elephant to ResNet-50 at various scales
ResNet was trained on 224x224 images
How much bigger can I make the image before the elephant is misclassified?
Rescaling Demo:
I tried rescales of [1.1, 1.5,3,5,10]
Elephant was correctly classified up till 5x scaling
Input size was 1120x1120
Confidence of classification decays slowly
At rescale factor of 10, ‘African Elephant’ is no longer in the top 3
Rescaling Demo:
Raw predictions:
Review: Rectified Linear Units (ReLU)
ReLU: max(0,x)
What’s the gradient in negative region?
Is there a problem?
Dying ReLU Problem
If input to ReLU is negative for the dataset, ReLU dies
Brief burst of research into addressing dying ReLUs
General idea is to have non-zero gradients even for negative inputs
Dead
Leaky ReLU & Parameterized ReLU
In Leaky ReLU, a is a hyperparameter.
In Parameterized ReLU, a is learned.
Djork-Arné Clevert, Thomas Unterthiner, Sepp Hochreiter Fast and Accurate Deep Network Learning by Exponential Linear Units
Exponential Linear Units (ELUs)
Djork-Arné Clevert, Thomas Unterthiner, Sepp Hochreiter Fast and Accurate Deep Network Learning by Exponential Linear Units
A tale of two papers...
Top right: paper that introduced PReLUs
Bottom right: paper that introduced Residual Networks
What do you notice about these papers?
Screenshots of both papers were taken from arXiv
Contextualizing
Same team that introduced PReLU created ResNet
Went back to ReLUs for ResNet
Focus shifted from activations to overall network design
Screenshots of both papers were taken from arXiv
Contextualizing
Same team that introduced PReLU created ResNet
Went back to ReLUs for ResNet
Focus shifted from activations to overall network design
Contextualizing
Same team that introduced PReLU created ResNet
Went back to ReLUs for ResNet
Focus shifted from activations to overall network design
In conclusion
The papers introducing each alternative activations claim they work well
ReLU still most popular
All the architectures we are about to discuss used ReLUs (and batch norm)
Djork-Arné Clevert, Thomas Unterthiner, Sepp Hochreiter Fast and Accurate Deep Network Learning by Exponential Linear Units
Stage 2:
“Understanding” ResNets
Review
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,�Deep Residual Learning for Image Recognition, CVPR 2016
Going deeper without residuals
Consider two non-residual networks
We call the 18 layer variant ‘plain-18’
We call the 34 layer variant ‘plain-34’
The ‘plain-18’ network outperformed `plain-34` on the validation set
Why do you think this was the case?
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,�Deep Residual Learning for Image Recognition, CVPR 2016
18 vs 34 layer ‘plain’ network
Vanishing gradients weren’t the issue
Overfitting wasn’t the issue
Representation power wasn’t the issue
Quote from ResNet paper:
We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN, which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,�Deep Residual Learning for Image Recognition, CVPR 2016
18 vs 34 layer ‘plain’ network
Vanishing gradients weren’t the issue
Overfitting wasn’t the issue
Representation power wasn’t the issue
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,�Deep Residual Learning for Image Recognition, CVPR 2016
Even the training error is higher with the 34 layer network
18 vs 34 layer ‘plain’ network
Vanishing gradients weren’t the issue
Overfitting wasn’t the issue
Representation power wasn’t the issue
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,�Deep Residual Learning for Image Recognition, CVPR 2016
18 vs 34 layer ‘plain’ network
Vanishing gradient wasn’t the issue
Overfitting wasn’t the issue
Representation power wasn’t the issue
? | ? | ? |
? | ? | ? |
? | ? | ? |
What should the weights be?
18 vs 34 layer ‘plain’ network
Vanishing gradient wasn’t the issue
Overfitting wasn’t the issue
Representation power wasn’t the issue
0 | 0 | 0 |
0 | 1 | 0 |
0 | 0 | 0 |
With ‘SAME’ padding, this will output the same feature map it receives as input
Optimization issues
Although identity is representable, learning it proves difficult for optimization methods
Intution: Tweak the network so it doesn’t have to learn identity connections
0 | 0 | 0 |
0 | 1 | 0 |
0 | 0 | 0 |
With ‘SAME’ padding, this will output the same feature map it receives as input
Optimization issues
Although identity is representable, learning it proves difficult for optimization methods
Intution: Tweak the network so it doesn’t have to learn identity connections
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,�Deep Residual Learning for Image Recognition, CVPR 2016
Optimization issues
Although identity is representable, learning it proves difficult for optimization methods
Intution: Tweak the network so it doesn’t have to learn identity connections
Result: Going deeper makes things better!
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,�Deep Residual Learning for Image Recognition, CVPR 2016
With residuals, the 34-layer network outperforms the 18 layer.
Optimization issues
Although identity is representable, learning it proves difficult for optimization methods
Intution: Tweak the network so it doesn’t have to learn identity connections
Result: Going deeper makes things better!
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,�Deep Residual Learning for Image Recognition, CVPR 2016
The architecture of the plain and residual networks were identical except for the skip connections
Interesting Finding
Less variation in activations for Residual Networks
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,�Deep Residual Learning for Image Recognition, CVPR 2016
Why do ResNets work? Some ideas:
They can be seen as implicitly ensembling shallower networks
They are able to learn unrolled iterative refinements
Can model recurrent computations necessary for recognition
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
Why do ResNets work? Some ideas:
They can be seen as implicitly ensembling shallower networks
They are able to learn unrolled iterative estimation
Can model recurrent computations necessary for recognition
Challenges the “representation view”
Image source: http://vision03.csail.mit.edu/cnn_art/
Why do ResNets work? Some ideas:
They can be seen as implicitly ensembling shallower networks
They are able to learn unrolled iterative estimation
Can model recurrent computations useful for recognition
Qianli Liao, Tomaso Poggio,Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex,
ResNets as Ensembles
Can think of ResNets as ensembling subsets of residual modules
With L residual modules there are 2L possible subsets of modules
If one of modules is removed, there are still 2L-1 possible subsets of modules
ResNets as Ensembles
Can think of ResNets as ensembling subsets of residual modules
With L residual modules there are 2L possible subsets of modules
If one of modules is removed, there are still 2L-1 possible subsets of modules
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
ResNets as Ensembles
Can think of ResNets as ensembling subsets of residual modules
With L residual modules there are 2L possible subsets of modules
If one of modules is removed, there are still 2L-1 possible subsets of modules
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
ResNets as Ensembles
Wanted to test this explanation
Tried dropping layers
Tried reordering layers
Found effective paths during training are relatively shallow
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
ResNets as Ensembles
Wanted to test this explanation
Tried dropping layers
Tried reordering layers
Found effective paths during training are relatively shallow
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
Dropping layers on VGG-Net is disastorous...
ResNets as Ensembles
Wanted to test this explanation
Tried dropping layers
Tried reordering layers
Found effective paths during training are relatively shallow
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
Dropping layers on ResNet is no big deal
ResNets as Ensembles
Wanted to test this explanation
Tried dropping layers
Tried reordering layers
Found effective paths during training are relatively shallow
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
Performance degrades smoothly as layers are removed
ResNets as Ensembles
Wanted to test this explanation
Tried dropping layers
Tried reordering layers
Found effective paths during training are relatively shallow
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
Though the total network has 54 modules; more than 95% of paths go through 19 to 35 modules
ResNets as Ensembles
Wanted to test this explanation
Tried dropping layers
Tried reordering layers
Found effective paths during training are relatively shallow
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
The Kendall Tau correlation coefficient measures the degree of reordering
ResNets as Ensembles
Wanted to test this explanation
Tried dropping layers
Tried reordering layers
Found effective paths during training are relatively shallow
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
[W]e show most gradient during training comes from paths that are even shorter, i.e., 10-34 layers deep.
Summary
ResNets seem to work because they facilitate the training of deeper networks
Are suprisingly robust to layers being dropped or reordered
Seem to be function approximations using iterative refinement
Stage 3:
Survey of Architectures
Recap (General Principles in NN Design)
Reduce filter sizes (except possibly at the lowest layer), factorize filters aggressively
Use 1x1 convolutions to reduce and expand the number of feature maps judiciously
Use skip connections and/or create multiple paths through the network
(Professor Lazebnik’s slides)
What are the current trends?
Some make minor modifications to ResNets
Biggest trend is to split of into several branches, and merge through summation
A couple architectures go crazy with branch & merge, without explicit identity connections
Identity Mappings in Deep Residual Networks�Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
What are the current trends?
Some make minor modifications to ResNets
Biggest trend is to split of into several branches, and merge through summation
A couple architectures go crazy with branch & merge, without explicit identity connections
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, Kaiming He. Aggregated Residual Transformations for Deep Neural Networks
What are the current trends?
ResNeXt
Inception ResNet
PolyNet
MultiResNet
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, Kaiming He. Aggregated Residual Transformations for Deep Neural Networks
Masoud Abdi, Saeid Nahavandi. Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks
Xingcheng Zhang, Zhizhong Li, Chen Change Loy, Dahua Lin. PolyNet: A Pursuit of Structural Diversity in Very Deep Networks
C. Szegedy et al., Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
What are the current trends?
Some make minor modifications to ResNets
Biggest trend is to split of into several branches, and merge through summation
A couple architectures go crazy with branch & merge, without explicit identity connections
C. Szegedy et al., Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Gustav Larsson, Michael Maire, Gregory Shakhnarovich FractalNet: Ultra-Deep Neural Networks without Residuals
Some try going meta..
Fractal of Fractals
Leslie N. Smith, Nicholay Topin, Deep Convolutional Neural Network Design Patterns
Residuals of Residuals
ResNet tweaks: Change order
Pre-activation ResNets
Same components as original, order of BN, ReLU, and conv changed
Idea is to have more direct path for input identity to propagate
Resulted in deeper, more accurate networks on ImageNet/CIFAR
Identity Mappings in Deep Residual Networks�Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
ResNet tweaks: Change order
Pre-activation ResNets
Same components as original, order of BN, ReLU, and conv changed
Idea is to have more direct path for input identity to propagate
Resulted in deeper, more accurate networks on ImageNet/CIFAR
Identity Mappings in Deep Residual Networks�Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
ResNet tweaks: Change order
Pre-activation ResNets
Same components as original, order of BN, ReLU, and conv changed
Idea is to have more direct path for input identity to propagate
Resulted in deeper, more accurate networks on ImageNet/CIFAR
ImageNet performance
Identity Mappings in Deep Residual Networks�Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
ResNet tweaks: Change order
Pre-activation ResNets
Same components as original, order of BN, ReLU, and conv changed
Idea is to have more direct path for input identity to propagate
Resulted in deeper, more accurate networks on ImageNet/CIFAR
CIFAR-10 performance
Identity Mappings in Deep Residual Networks�Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
ResNet tweaks: Wide ResNets
Use pre-activation ResNet’s basic block with more feature maps
Used parameter “k” to encode width
Investigated relationship between width and depth to find a good tradeoff
Sergey Zagoruyko, Nikos Komodakis Wide Residual Networks
ResNet tweaks: Wide ResNets
Use pre-activation ResNet’s basic block with more feature maps
Used parameter “k” to encode width
Investigated relationship between width and depth to find a good tradeoff
Sergey Zagoruyko, Nikos Komodakis Wide Residual Networks
ResNet tweaks: Wide ResNets
Use pre-activation ResNet’s basic block with more feature maps
Used parameter “k” to encode width
Investigated relationship between width and depth to find a good tradeoff
Sergey Zagoruyko, Nikos Komodakis Wide Residual Networks
ResNet tweaks: Wide ResNets
Use pre-activation ResNet’s basic block with more feature maps
Used parameter “k” to encode width
Investigated relationship between width and depth to find a good tradeoff
Sergey Zagoruyko, Nikos Komodakis Wide Residual Networks
ResNet tweaks: Wide ResNets
These obtained state of the art results on CIFAR datasets
Were outperformed by bottlenecked networks on ImageNet
Best results on ImageNet were obtained by widening ResNet-50
Sergey Zagoruyko, Nikos Komodakis Wide Residual Networks
ResNet tweaks: Wide ResNets
These obtained state of the art results on CIFAR datasets
Were outperformed by bottlenecked networks on ImageNet
Best results on ImageNet were obtained by widening ResNet-50
Sergey Zagoruyko, Nikos Komodakis Wide Residual Networks
ResNet tweaks: Wide ResNets
These obtained state of the art results on CIFAR datasets
Were outperformed by bottlenecked networks on ImageNet
Best results on ImageNet were obtained by widening ResNet-50
Sergey Zagoruyko, Nikos Komodakis Wide Residual Networks
“With widening factor of 2.0 the resulting WRN-50-2-bottleneck outperforms ResNet- 152 having 3 times less layers, and being significantly faster.”
Aside from ResNets
FractalNet and DenseNet
FractalNet
A competitive extremely deep architecture that does not rely on residuals
Gustav Larsson, Michael Maire, Gregory Shakhnarovich
FractalNet: Ultra-Deep Neural Networks without Residuals
FractalNet
A competitive extremely deep architecture that does not rely on residuals
Interestingly, its architecture is similar to an unfolded ResNet
Gustav Larsson, Michael Maire, Gregory Shakhnarovich
FractalNet: Ultra-Deep Neural Networks without Residuals
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, arxiv 2016
DenseNet (Within a DenseBlock)
Every layer is connected to all other layers.
For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs�into all subsequent layers.
DenseNet
Alleviate the vanishing-gradient problem.
Strengthen feature propagation.
Encourage feature reuse.
Substantially reduce the number of parameters.
Concluding Remarks
At surface level, there’s tons of new architectures that are very different
Upon closer inspection, most of them are reapplying well established principles
Universal principles seem to be having shorter subpaths through the networks
Identity propagation (Residuals, Dense Blocks) seem to make training easier
References
Sergey Ioffe,Christian Szegedy,Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, http://jmlr.org/proceedings/papers/v37/ioffe15.pdf
K. He, X. Zhang, S. Ren, and J. Sun, Deep Residual Learning for Image Recognition, CVPR 2016 https://arxiv.org/abs/1512.03385
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, Identity Mappings in Deep Residual Networks https://arxiv.org/abs/1603.05027
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, Kaiming He, Aggregated Residual Transformations for Deep Neural Networks, https://arxiv.org/pdf/1611.05431v1.pdf
Max Jaderberg, Karen Simonyan, Andrew Zisserman, Koray Kavukcuoglu: Spatial Transformer Networks https://arxiv.org/abs/1506.02025
Leslie N. Smith, Nicholay Topin, Deep Convolutional Neural Network Design Patterns, https://arxiv.org/abs/1611.00847
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, Alex Alemi: Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning https://arxiv.org/abs/1602.07261
Gustav Larsson, Michael Maire, Gregory Shakhnarovich, FractalNet: Ultra-Deep Neural Networks without Residuals https://arxiv.org/abs/1605.07648
Gao Huang, Zhuang Liu, Kilian Q. Weinberger, Laurens van der Maaten: Densely Connected Convolutional Networks https://arxiv.org/pdf/1608.06993v3.pdf
Rupesh Kumar Srivastava, Klaus Greff, Jürgen Schmidhuber: Highway Networks https://arxiv.org/abs/1505.00387
Xingcheng Zhang, Zhizhong Li, Chen Change Loy, Dahua Lin, PolyNet: A Pursuit of Structural Diversity in Very Deep Networks, https://arxiv.org/abs/1611.05725
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. https://arxiv.org/abs/1412.6806
Andreas Veit, Michael Wilber, Serge Belongie, Residual Networks Behave Like Ensembles of Relatively Shallow Networks, https://arxiv.org/pdf/1605.06431v2.pdf
Klaus Greff, Rupesh K. Srivastava & Jürgen Schmidhuber, Highway and Residual Networks Learn Unrolled Iterative Estimation,https://arxiv.org/pdf/1612.07771v1.pdf
Min Lin, Qiang Chen, Shuicheng Yan, Network In Network, https://arxiv.org/abs/1312.4400
Brian Chu, Daylen Yang, Ravi Tadinada, Visualizing Residual Networks, https://arxiv.org/abs/1701.02362
Djork-Arné Clevert, Thomas Unterthiner, Sepp Hochreiter: Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) https://arxiv.org/abs/1511.07289
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, https://arxiv.org/abs/1502.01852
Anish Shah, Eashan Kadam, Hena Shah, Sameer Shinde, Sandip Shingade, Deep Residual Networks with Exponential Linear Unit, https://arxiv.org/abs/1604.04112