Deep Networks

(DenseNet & ResNet)

Rowel Atienza

rowel@eee.upd.edu.ph

University of the Philippines

DenseNet

Huang, Gao, et al. "Densely connected convolutional networks Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

Problem of

Vanishing Gradient

Gradients vanish when computed at shallow layers due to the nature of backprop that uses chain rule

A variable that is dependent on another variable that is small is also small. This smallness propages to the shallow layers until it vanishes.

DenseNet (>100 layers)

Solution

• Encourage information flow down to shallow layers
• How do we maximize information flow between networks?
• DenseNet: Connect all layers directly with each other
• DenseNet: Feature maps are combined by concatenation

​

DenseNet Input

DenseNet

DenseNet vs ResNet

DenseNet 2 Issues to Address

• How to address feature maps growth rate?
• How to reduce feature map size?

Feature Maps Growth Rate

BottleNeck Layer Controls Growth Rate

Transition Layer Bridges Blocks w/ Different Feature Map Sizes

• Deep DenseNet with decreasing feature map size are divided into to Dense Blocks
• A transition layer between 2 Dense Blocks block enables the change of feature map size;
• Transition Layer = Convolution and Pooling

CIFAR10

10 categories

50k train set

10k test set

100-layer DenseNet Architecture

(Accuracy is > 93.55%)

DenseNet on CIFAR10

ResNet v1 & v2

He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016a.

He, Kaiming, et al. "Identity mappings in deep residual networks." European Conference on Computer Vision. Springer International Publishing, 2016b.

RESNET

• Problem of vanishing gradients/exploding gradients
• In theory, as the network gets deeper, it gets better since it learns more representations

RESNET

• However, this is not the case for plain networks
• In more common cases, as the network has maximized its performance, it starts to degrade as the training continues
• The problem is not with overfitting

​

RESNET

ResNet Residual

Block

ResNet v1 on CIFAR 10

ResNet

on

CIFAR10

ResNet v2

Accuracy of ResNet v1/v2 on CIFAR10

END