Efficient Visual Self-Attention

Efficiency, Generalization, and Paradigm Shift

Shen Zhuoran, Nov. 1, 2021

Overview

• Efficient attention: from quadratic to linear complexities.
• Global context module: efficient streaming attention for videos.
• Global self-attention network: going all-in on attention.

Motivation for attention

• Long-range dependency modeling.
• Content-adaptive connections.

Dot-product attention

Efficient attention

Efficient attention

Interpretation

Empirical comparison vs. the non-local module

Additional results

Stereo depth estimation (Scene Flow)

Temporal action localization (THUMOS14)

Global context module

Semi-supervised video object segmentation

Deep learning approaches

• Online learning
• Accurate but slow
• Offline learning
• Template matching
• Uses first frame as template, prone to appearance change
• Mask propagation
• Propagates the mask from previous frame, prone to error accumulation
• Hybrid methods

Space-time memory module

Global context module

GC module vs. STM module

Empirical results

Visualizations

Global Self-Attention Networks

Motivations for fully-attentional modeling

• Elimination of semanticity-resolution trade-off
• Representation learning
• Ability to simulate convolution (Attention vs. CNN)

Goal

Efficient attention

• Lack of spatial information.

21

Encoding positional information

22

Encoding relative positions for self-attention

23

Axial attention

24

GSA module

25

GSA-Net

26

Results on ImageNet

27

Discussion

• Attentional networks exhibit higher capacity.
• Attentional networks learn faster.
• Axial attention is slow for batched inference on accelerators.

28

Thank you