Understanding The Robustness In Vision Transformers

Daquan Zhou, Zhiding Yu, Enze Xie

Chaowei Xiao, Anima Anandkumar, Jiashi Feng and Jose M. Alvarez

Advances in Visual Recognition

Larger Models

Faster Computing

Bigger Data

Standard Visual Recognition Is Getting Saturated

ImageNet-1K

Top Performing Models

Challenge – Real World Data Are Imperfect

• Domain shift
• Data noise
• Imbalanced distribution
• Can contain Occlusions
• Can be Cluttered
• Can be Ambiguous
• Can be Deceiving
• •••

Corrupted ImageNet (ImageNet-C)

COCO-C/Cityscapes-C

More Challenging Scenarios

Hendrycks et al., Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, ICLR19

ResNet

Image Classification

Semantic Segmentation

How Well Do Current DNNs Perform?

ViTs Are Robust Learners

Bai et al., Are Transformers More Robust Than CNNs? NeurIPS21

Naseer et al., Intriguing Properties of Vision Transformers, NeurIPS21

Mao et al., RVT: Towards Robust Vision Transformer, CVPR22 

Zhang et al., Delving Deep into the Generalization of Vision Transformers under Distribution Shifts, CVPR22 

Delving Deeper into ViT’s Robustness

Visual Grouping and Information Bottleneck

“I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnesses and nuances of colour. Do I have "327"? No. I have sky, house, and trees.”

� ——Max Wertheimer

Visual Grouping

Information Bottleneck (IB)

“Information bottlenecks are extremely interesting. I have to listen to it ten thousand times to really understand it. It's hard to hear such original ideas today. Maybe it's the key to the puzzle.”

——Geoffrey Hinton

Visual Grouping

Spectral Clustering vs. Self-Attention

Image Credit: Jay Alammar, The Illustrated Transformer.

Image Credit: Spectral Clustering for Molecular Emission Segmentation.

Emerging Properties in ViTs

Caron et al., Emerging Properties in Self-Supervised Vision Transformers, ICCV21

Correlation between grouping and robustness over network blocks

The Trinity among Grouping, IB and Robust Generalization

MSHA as Mixture of IBs

Fully Attentional Network

• Further deploy the attention mechanism reinforce the clustering phenomenon
• Fore-ground objects are better captured
• Directly apply SA along the channel dimension has two drawbacks
• Large computational overhead
• Low parameter efficiency

Main Results – Image Classification

Main Results – Downstream Tasks

https://github.com/NVlabs/FAN

Code Available