1 of 21

Pre-trained CNNs

Prof. Seungchul Lee

Industrial AI Lab.

2 of 21

ImageNet

  • Human performance = 5.1 %

2

from Kaiming He slides "Deep residual learning for image recognition," ICML, 2016.

3 of 21

ImageNet

3

4 of 21

LeNet

  • CNN = Convolutional Neural Networks = ConvNet
  • LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition.
  • All are still the basic components of modern ConvNets!

4

Yann LeCun

5 of 21

AlexNet

  • Simplified version of Krizhevsky, Alex, Sutskever, and Hinton. "Imagenet classification with deep convolutional neural networks." NIPS 2012

​

  • LeNet-style backbone, plus:
    • ReLU [Nair & Hinton 2010]
      • “RevoLUtion of deep learning”*
      • Accelerate training
    • Dropout [Hinton et al 2012]
      • In-network ensembling
      • Reduce overfitting
    • Data augmentation
      • Label-preserving transformation
      • Reduce overfitting

5

6 of 21

VGG-16/19

  •  

6

7 of 21

GoogleNet/Inception

  • Multiple branches
    • e.g., 1x1, 3x3, 5x5, pool
  • Shortcuts
    • stand-alone 1x1, merged by concatenation
  • Bottleneck
    • Reduce dim by 1x1 before expensive 3x3/5x5 conv

7

Inception module

8 of 21

ResNet (Deep Residual Learning)

  •  

8

  • If identity were optimal, easy to set weights as 0

​

  • If optimal mapping is closer to identity, easier to find small fluctuations

9 of 21

DensNets

  • Densely Connected Convolutional Networks

9

Huang, Gao, et al., “Densely connected convolutional networks”

Proceedings of the IEEE conference on computer vision and pattern recognition. Vol. 1. No. 2. 2017.

10 of 21

U-Net

  • The U-Net owes its name to its symmetric shape
    • better segmentation in medical imaging

​

10

Ronneberger, Olaf; Fischer, Philipp; Brox, Thomas (2015),

“U-Net: Convolutional Networks for Biomedical Image Segmentation.” arXiv:1505.04597

11 of 21

Modern CNNs

  • LeNet
  • AlexNet
  • VGG
  • GoogleNet/Inception
  • ResNet
  • DensNet
  • U-Net

11

12 of 21

Pre-trained Models

  • Model Selection

​

​

​

​

​

​

​

  • Model Summary

​

​

12

13 of 21

Pre-trained Models

  • Input Resize

​

​

​

  • Model Prediction

13

14 of 21

Transfer Learning

Prof. Seungchul Lee

Industrial AI Lab.

15 of 21

Pre-trained Models

  • Training a model on ImageNet from scratch takes days or weeks.
  • Many models trained on ImageNet and their weights are publicly available!
  • Transfer learning
    • Use pre-trained weights, remove last layers to compute representations of images
    • The network is used as a generic feature extractor
    • Train a classification model from these features on a new classification task
    • Pre- trained models can extract more general image features that can help identify edges, textures, shapes, and object composition
    • Better than handcrafted feature extraction on natural images

15

16 of 21

Image Classification with VGG16

  • Target data (usually small size)
    • 5 classes

​

​

16

17 of 21

Image Classification with VGG16

  • Target data (usually small size)
    • 5 classes

​

​

  • Target data to VGG16

17

18 of 21

Image Classification with VGG16

  • Target data (usually small size)
    • 5 classes

​

​

  • Target data to VGG16
    • Poor performance

18

19 of 21

Transfer Learning

  • We assume that these model parameters contain the knowledge learned from the source data set and that this knowledge will be equally applicable to the target data set.
  • We will train the output layer from scratch, while the parameters of all remaining layers are fine tuned based on the parameters of the source model.
  • Or initialize all weights from pre-trained model, then train them with target data

​

19

Random

initialization

Copy

Copy

Output layer

Layer L-1

Layer 1

Target data

Target

model

Output layer

Layer L-1

Layer 1

Source data

Source

model

Pre-train

Train from

scratch

Fine-tune or fixed

Image from http://d2l.ai/

20 of 21

Transfer Learning Structure and Implementation

20

Non-trainable

trainable

21 of 21

Testing

21