Transformers

in Computer Vision

Seyed Iman Mirzadeh

​

Introduction

• Architectural progress in Deep Learning is very slow.
• We are still using mostly MLPs, CNNs, and RNNs from 1990s.
• In 2017, Transformers came and became the de facto standard in NLP (almost no RNN is used anymore in NLP).
• It is interesting to see if they are useful in computer vision, too.

Background: Transformers

Image from: http://jalammar.github.io/illustrated-transformer/

Vision Transformer (ViT)

How does ViT Work?

Link

Wait!

Couldn’t anyone else think of breaking images to patches?

Yes. But that’s not the magic of ViT. The magic comes from pre-training large models on ridiculously gigantic datasets!

The model is pre-trained from multiple large scale datasets with deduplication to support fine tuning (smaller dataset) downstream tasks.

• ILSVRC-2012 ImageNet dataset with 1k classes and 1.3M images
• ImageNet-21k with 21k classes and 14M images
• JFT with 18k classes and 303M high-resolution images

​

Results

Results (2)

Punchline of the Paper

• In over-parametrized training regime on with pre-training on ridiculously large datasets, It doesn’t matter if we use CNNs or Transformers.

Why did I choose this paper?

• I personally believe transformers are not good and some architecture to CNNs will be better for vision tasks.

​

• But this is a win-win scenario for me in a long run:
• Either I’m wrong, and It’s interesting to find out how researchers will overcome the challenges to make transformers better than CNNs.
• Or I’m right, then l learn how researchers [unintentionally] sell their wrong ideas! (might be a useful skill!)

ViT on Smaller datasets

Source: https://github.com/ra1ph2/Vision-Transformer

Transformers in Vision: Other Attempts

DETR: Results