Vision Transformers

CS5670: Computer Vision

Announcements

• Final deliverables: Project 5 (Parts A & B, Part A due tomorrow)
• Final exam, in class May 6

​

• Course evaluations coming up, worth a small amount of extra credit

Readings

• Szeliski 2^nd Edition, Chapter 5.5.3
• Foundations of Computer Vision
• https://visionbook.mit.edu/transformers.html

Recall: ConvNets

Elgendy, Deep Learning for Vision Systems, https://livebook.manning.com/book/grokking-deep-learning-for-computer-vision/chapter-5/v-3/

ConvNets assume spatial locality

• Assume nearby pixels are more important to making decisions than far away pixels (an example of an “inductive bias”)
• Only after stacking together several convolutional layers with spatial downsampling can distant pixels “talk” to each other
• As image datasets grow, we can do better by removing the spatial locality assumption and learning how to process images from scratch

​

An alternative to convolution: Attention

• Goal: consider long-range relationships between pixels

An alternative to convolution: Attention

Step 1: Break image into patches

An alternative to convolution: Attention

Step 1: Break image into patches

An alternative to convolution: Attention

Step 2: Map each patch to three vectors:

Query (Q), Key (K), and Value (V)

linear mappings

Q₁

K₁

V₁

An alternative to convolution: Attention

Q₁

linear mappings

K₁

V₁

Q₅

K₅

V₅

Step 3: For each patch, compare its query vector to all key vectors

An alternative to convolution: Attention

Step 3: For each patch, compare its query vector to all key vectors

An alternative to convolution: Attention

Step 4: Compute weighted sum of value vectors

An alternative to convolution: Attention

Step 5: Repeat for all patches

An alternative to convolution: Attention

Result: we’ve transformed all of the input patches into new vectors, by comparing vectors derived from all pairs of patches

This operation is called attention – the network can choose, for each patch, which other patches to attend to (i.e., give high weight to)

Unlike convolution, a patch is allowed to talk to the entire image

Attention is a set-to-set operation – it is equivariant to permuting the patches

An alternative to convolution: Attention

Parameters: weight matrices W_q, W_k, W_v that map input patches to query, key, and value vectors

Details

• Rather than working with raw RGB image patches, the patches can themselves be features (e.g., produced by a linear mapping from RGB patches, or the output of a CNN). (These vectors are often called tokens)
• The feature vectors produced by the attention layer are often passed through an MLP (adding more parameters to the system)
• Each patch can be combined with a positional encoding indicating the spatial location of the patch, enabling spatial reasoning
• Instead of single W_q, W_k, W_v weight matrices, multiple linear mappings can be learned for an attention layer, and the resulting features concatenated (multi-headed attention)

Transformers

• Just like any network layer, we can stack attention layers – the output of one becomes the input to the next – to form a bigger network, called a transformer
• Transformers are very large, powerful learners that transcend convolutional networks by representing a larger class of functions

Vision Transformer (ViT)

• The network defined so far is designed for image classification, and roughly follows:

ICLR 2021

Vision Transformer (ViT)

How is the output class computed?

Vision Transformer (ViT)

How is the output class computed?

At the time, outperformed CNN-based approaches on image classification tasks

Vision Transformer (ViT)

• Note: this is just one possible approach – lots of others variants of transformers for vision task exist!
• (For instance, combinations of transformers and CNNs)

DPT: Dense Prediction Transformers�[Ranftl et al., 2021]

• Predicts an image-shaped output (e.g., segmentation map or depth map) from an image-shaped input

DPT architecture

Reassemble operation

Fusion operation

DPT: Depth prediction results

Input

MiDaS (CNN-based)

DPT (Transformer)

DPT: Attention maps

Input

Depth prediction

Attention maps for upper right corner

Attention maps for lower right corner

Questions?

Other types of transformers

Autoregressive transformer methods

• Common for LLMs
• Outputs are generated one by one, where each output (word, patch, etc) is fed back to the network to generate the next

“Who was the 16^th President of the United States?”

Condensed input representation

“Who was the 16^th President of the United States?”

[Empty]

Token is fed back into the decoder, next token predicted, etc. Some randomness is used to select each next token.

Also called a “sequence to sequence model”

Also works for images

Also works for images

Idea: generate an image by producing each (tokenized) patch in raster-scan order

Parti Text-to-image model

Cross-attention between text prompt and generated image patches

Text comes in:

Image patches come out in raster order

Autoregressive models

• Why output tokens one by one instead of all at once?
• Answer: autoregressive models are better for generative tasks like text-to-image where there are many correct answer
• (Diffusion models are also good for such tasks)
• Autoregressive methods can lead to more diverse, consistent, and high-quality outputs in generative tasks
• (An important component in the process is some randomness.)

Non-autoregressive models

• Output all tokens at once (e.g., DPT)
• Well-suited for when the output is more deterministic (“one correct answer”)
• Example: view interpolation

Large View Synthesis Model (LVSM)

Input Views &

Their Poses

LVSM

Synthesized

Target View

Target View Pose

…

`

Target Tokens

Input

Tokens

Updated

Target Tokens

Large View Synthesis Model (LVSM)

Input Views &

Their Poses

LVSM

Synthesized

Target View

Target View Pose

…

`

Target Tokens

Input

Tokens

Updated

Target Tokens

Tokenize & Detokenize

Input Views &

Plücker Rays

Target View

Plücker Rays

…

LVSM Results

CNNs vs Transformers

• Transformers have been thought of as more scalable and more modern

Questions?