1 of 13

Multi-criteria Token Fusion with One-step-ahead Attention

for Efficient Vision Transformers

Sanghyeok Lee, Joonmyung Choi, Hyunwoo J. Kim

​

Korea University

Korea Univ.

1

CVPR 2024

2 of 13

Motivation

  • ViTs

Total Complexity

​

​

​

​

​

-> Quadratic cost in the number of tokens and channels!

Korea Univ.

2

CVPR 2024

3 of 13

  • Previous approaches

​

    • local self-attention within predefined windows

​

    • Introducing light-weight attention mechanism

​

    • Token reduction (Token pruning / fusion)

​

      • Utilizing existing ViTs without altering their architecture!

​

​

​

Motivation

Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." ICCV, 2021.

Chu, Xiangxiang, et al. "Twins: Revisiting the design of spatial attention in vision transformers." NeurIPS, 2021.

Wang, Sinong, et al. "Linformer: Self-attention with linear complexity.” Arxiv, 2020.

Xiong, Yunyang, et al. "Nyströmformer: A nyström-based algorithm for approximating self-attention." AAAI, 2021.

Korea Univ.

3

CVPR 2024

4 of 13

  • Token reduction (Informativeness-based)

​

    • Pruning/fusing uninformative tokens

EViT (ICLR ‘22)

DynamicViT (NeurIPS ‘21)

Motivation

Problem : informativeness-based fusion often merges substantially dissimilar tokens, resulting in collapsed representations

Korea Univ.

4

CVPR 2024

5 of 13

  • Token reduction (Similarity-based)

​

    • Pruning/fusing to reduce redundancies in tokens.

Token pooling (WACV ‘23)

ToMe (Token merging) (ICLR ’23, Oral)

Motivation

Problem : similarity-based token fusion is prone to combine the foreground tokens despite the importance of it.

Multi-Criteria Token Fusion (MCTF)

to optimize vision transformers by fusing tokens based on multi-criteria.

Korea Univ.

5

CVPR 2024

6 of 13

  • Attraction function & multi-criteria

MCTF

: attraction function by k-th critertion

: temperature parameter to adjust the influence of k-th criteron

Korea Univ.

6

CVPR 2024

7 of 13

  • Attraction function & multi-criteria

​

    • Similarity : Remove redundancy

​

​

​

2. Informativeness : Capture importance

​

​

​

3. Size : Suppress large-sized tokens

​

​

​

MCTF

Excessive fusion of the foreground object

the tokens in the background are largely fused

Korea Univ.

7

CVPR 2024

8 of 13

  • One-step-ahead attention

​

MCTF

low Informativeness in previous layer

But High informativeness in the next layer

Korea Univ.

8

CVPR 2024

9 of 13

  • Token reduction consistency

​

    • Different

> different representation

(Token-level augmentation)

​

    • Select two reduction ratio

(Fixed, dynamic)

    • Update the model with

​

​

​

​

​

MCTF

Korea Univ.

9

CVPR 2024

10 of 13

  • Comparison with previous Token reduction Method

​

​

​

Experiments

Korea Univ.

10

CVPR 2024

11 of 13

  • MCTF with/without training

​

Experiments

Korea Univ.

11

CVPR 2024

12 of 13

  • Qualitative results

Experiments

Korea Univ.

12

CVPR 2024

13 of 13

Multi-criteria Token Fusion with One-step-ahead Attention

for Efficient Vision Transformers

Korea Univ.

13

CVPR 2024