DualCoOp: Fast Adaptation to �Multi-Label Recognition with Limited Annotations

Ximeng Sun¹, Ping Hu¹, Kate Saenko^1,2

^{1: Boston University, 2: MIT-IBM Watson AI Lab}

​

^{NeurIPS 2022}

​

Multi-label recognition

• A harder task than standard multiclass classification
• Difficult to annotate exhaustively
• Many images have only partial labels: some categories are not annotated

​

DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations. Sun, Hu, and Saenko, NeurIPS’22

Zero-shot multi-label recognition

• Given labeled training images of N “known” categories
• Fully and partially labeled

car=?

dog=?

person=?

. . .

• Predict if novel objects (unseen at training) are present in the input image

. . .

DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations. Sun, Hu, and Saenko, NeurIPS’22

CLIP

PRO: powerful zero-shot model

​

CON: not designed for multilabel classification

DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations. Sun, Hu, and Saenko, NeurIPS’22

Our approach: DualCoOp

• Harness powerful pretrained V&L model
• Key idea: learn dual prompts (positive & negative) for the text encoder of CLIP

DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations. Sun, Hu, and Saenko, NeurIPS’22

p(“Animal”)

DualCoOp is light-weight and efficient

Prior work trains model parameters, while we only learn low-dimensional prompts

DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations. Sun, Hu, and Saenko, NeurIPS’22

DualCoOp: zero-shot multilabel classification

This result and attention map shows that our model can focus even on small objects

DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations. Sun, Hu, and Saenko, NeurIPS’22

DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations. Sun, Hu, and Saenko, NeurIPS’22

DualCoOp:Example zero-shot results

DualCoOp outperforms prior work while learning a fraction of parameters

• Higher F1 on zero-shot setting (test only on unseen classes) and generalized zero-shot setting (test on both seen and unseen classes)
• See paper for more results on NUS-WIDE etc.

​

DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations. Sun, Hu, and Saenko, NeurIPS’22