1 of 20

Popular (pre-trained) Convolution Neural Network (CNN) architecture choices

2 of 20

References

  • Top 10 CNN Architectures Every Machine Learning Engineer Should Know, by Trung Anh Dang [ Link ]
  • Illustrated: 10 CNN Architectures by Raimi Karim [ Link ]

3 of 20

CNN architecture timeline

Source: https://towardsdatascience.com/illustrated-10-cnn-architectures-95d78ace614d

4 of 20

Legends used in this presentation

5 of 20

LeNet-5

  • One of the earliest (1998). Guess who were the inventor(s)?
    • Yann LeCun, Leon Bottou, Yoshua Bengio, Patrick Haffner
  • It has 5 layers [2 Conv, 3 fully connected layers]. 60,000 Learnable parameters
  • Proved to be effective in classifying handwritten digits [MNIST dataset]

6 of 20

LeNet-5

7 of 20

AlexNet

  • Introduced in 2012. Guess who were the inventors?
    • Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton
  • It has 8 layers [5 Conv + 3 FC]: 60M learnable parameters.
  • First implementation of ReLU and dropout features.

8 of 20

AlexNet

  • Introduced in 2012. Guess who were the inventors?
    • Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton
  • It has 8 layers [5 Conv + 3 FC]: 60M learnable parameters.
  • First implementation of ReLU and dropout features.

9 of 20

VGG-16

  • Introduced in 2014 by the Visual Geometry Group at Univ of Oxford.
  • It has 13 CONV + 3 FC layers. 138M learnable parameters.
  • A pre-trained VGG-16 (or later variants) are now used to extract features from images [Transfer Learning]

10 of 20

Inception – V1 / GoogleNet

  • Introduced in 2014 by two research groups [Univ of Michigan, Univ of North Carolina Chapel Hill, and Google]
  • Building networks using modules/blocks. Instead of stacking convolutional layers, the authors stacked modules or blocks, within which are convolutional layers.
    • How does that relate to the naming? Any hint?
  • 5M learnable parameters

11 of 20

Inception – V1

12 of 20

  • Later the architecture became so popular there were pressing need to bring sequels:
    • Inception-v2 (2016)
    • Inception-v3 (2016)
    • Inception-v4 (2017)

13 of 20

ResNet-50

  • Introduced in 2015 by Microsoft Research.
    • The skip connections
    • First to adapt batch normalization.
  • 26M learnable parameters

14 of 20

ResNet-50

15 of 20

Xception

  • Introduced in 2016 (by Google)
  • Revised/improved Inception-v1

16 of 20

Xception

17 of 20

Inception-ResNet-v2

  • Introduced in 2016 (by Google)

18 of 20

Inception-ResNet-v2

19 of 20

ResNet-Xt

  • Introduced in 2017 (Facebook Research, UCSD)

20 of 20

Summary