1 of 22

Jihwan Park, Chanhyeong Yang, Jinyoung Park, Taehoon Song, Hyunwoo J. Kim*

RegFormer: Transferable Relational Grounding for Efficient Weakly-Supervised Human-Object Interaction Detection

1

Korea University

MLV Lab

NeurIPS 2024

2 of 22

Human-Object Interaction Detection

    • Localize human & object and classify the interactions between them
    • Only the interactive instances should be detected

HOI triplet: {human1, ride ,bicycle3}, {human1, hold, umbrella2}

1

2

3

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

3 of 22

Human-Object Interaction Detection

  • HOI detection requires triplet annotations for training: [ (human box), (action category), and (object box, category) ].

 

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

4 of 22

Weakly Supervised Learning

  • Weakly supervised learning becomes a practical alternative.

 

HOI recognition model

 

 

 

Available annotations

​

Person ride bicycle : 0.8

Person hold bicycle : 0.9

Person eat apple : 0.1

Person wear jacket : 0.8

Person hold banana : 0.1

​

HOI classification

Classification

Loss

 

Training

​

Object

Detector

: ride, hold

HOI detection

1

2

1

2

Inference

Pairing

HOI recognition model

Transfer

Image-level supervision

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

5 of 22

Weakly Supervised Learning

  • Since detector outputs all the possible human and object, HOI recognition model should recognize truly interactive ones, while process efficiently

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

6 of 22

Weakly Supervised Learning

  • Union regions include surrounding irrelevant instances, misleading the classifier on non-interactive pairs.
  • False positives from irrelevant regions

 

  • Computational overhead
  • Instance features tied to a specific detector; requires retraining when the detector changes.
  • Detector-classifier coupling

HOI recognition model

Cropped union

region images

Batch calculation.. Two slow

4

3

1

2

1

Detection results

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

7 of 22

Weakly Supervised Learning

  • Union regions include surrounding irrelevant instances, misleading the classifier on non-interactive pairs.
  • False positives from irrelevant regions

 

  • Computational overhead
  • Instance features tied to a specific detector; requires retraining when the detector changes.
  • Detector-classifier coupling

HOI recognition model

Cropped union

region images

Association

Prediction scores (riding)

1

1

1

3

1

2

1

4

0.648

0.645

0.622

0.635

TP

FP

FP

FP

4

3

1

2

1

Detection results

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

8 of 22

Weakly Supervised Learning

  • Union regions include surrounding irrelevant instances, misleading the classifier on non-interactive pairs.
  • False positives from irrelevant regions

 

  • Computational overhead
  • Instance features tied to a specific detector; requires retraining when the detector changes.
  • Detector-classifier coupling

Detector A

HOI recognition model

 

Detector B

Direct Replace ❌

 

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

9 of 22

Weakly Supervised Learning

  • Union regions include surrounding irrelevant instances, misleading the classifier on non-interactive pairs.
  • False positives from irrelevant regions

 

  • Computational overhead
  • Instance features tied to a specific detector; requires retraining when the detector changes.
  • Detector-classifier coupling

Detector A

HOI recognition model

 

Detector B

Direct Replace ❌

 

  • Irrelevant instances hinder accurate prediction, leading to numerous false positives

Relational Grounding Transformer (RegFormer)

lightweight interaction recognition module that seamlessly transfers from image-level to instance-level HOI reasoning

without additional training.

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

10 of 22

Overall pipeline – image-level training

person-swing-baseball bat

person-swing-baseball bat

HOI triplet classes

person-swing-baseball bat

Multi-label recognition model

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

11 of 22

Overall pipeline – instance-level inference

​

Object Detector

Detection results

No instance-level HOI annotations

or additional training required!

Multi-label recognition model

HOI triplet tuple

plug

play

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

12 of 22

Sequential process

H-O classes

person-baseball bat

Query generator

Visual�Encoder

 

A photo of �a baseball bat

A photo of �a baseball bat

A photo of a person�swinging an object

Action classes

Text Encoder

 

 

 

 

 

swing

hold

eat

0.9

0.8

0.1

Object Detector

Interactiveness score

Interactiveness scoring

Interaction Decoder

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

13 of 22

Spatially Grounded pairwise Query

Visual�Encoder

Object classes

Text Encoder

 

 

 

 

Patch-level Similarity

 

Human class

A photo of

a person

A photo of �a baseball bat

A photo of �a baseball bat

A photo of �a baseball bat

 

MLP

 

 

Patch-wise Softmax

Pairwise instance encoder

Image-level

patch importance score

 

 

 

 

Aggregation

Projection

 

 

 

 

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

14 of 22

Spatially Grounded pairwise Query

Visual�Encoder

Object classes

Text Encoder

 

 

 

 

Patch-level Similarity

 

Human class

A photo of

a person

A photo of �a baseball bat

A photo of �a baseball bat

A photo of �a baseball bat

 

MLP

 

 

Patch-wise Softmax

Object Detector

Region-aware Masking

 

 

Detection results

 

 

Instance-level

patch importance score

Aggregation

Projection

 

 

Pairwise Instance Encoder

 

 

 

 

 

 

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

15 of 22

Interactiveness-aware learning

 

 

 

 

 

 

 

 

 

 

 

 

Score Aggregation

Interaction Decoder

A photo of �a baseball bat

A photo of �a baseball bat

A photo of a person�swinging an object

Action classes

Text Encoder

 

 

 

 

 

 

 

 

 

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

16 of 22

Interactiveness-aware learning

 

 

Score Aggregation

 

 

 

 

Global interactiveness score

 

 

Score Aggregation

 

 

Local interactiveness score

 

 

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

17 of 22

Comparison with HOI detectors

Standard HICO-DET

Zero-Shot HICO-DET

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

18 of 22

Component & Efficiency Analysis

[🔍 Component] Each component consistently improves performance; combined yields +12.52 Full mAP over ML-Decoder.

X 120 faster!

[⏱️ Efficiency] RegFormer enables single backbone forward; about 120 times faster than ML-Decoder.

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

19 of 22

Analysis on Interactivenes-aware Learning

[🔍 Local vs Global] Local and global interactiveness are complementary; global corrects inflated local scores on non-interactive regions.

[🔍 Path importance] IA provides explicit supervision that enables accurate localization of multiple interactive instances in dense scenes, where spatial grounding alone fails.

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

20 of 22

Analysis on Interactivenes-aware Learning

[🔍 Local vs Global] Local and global interactiveness are complementary; global corrects inflated local scores on non-interactive regions.

[🔍 Path importance] IA provides explicit supervision that enables accurate localization of multiple interactive instances in dense scenes, where spatial grounding alone fails.

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

21 of 22

Qualitative results

MLV Lab

Korea University

MLV Lab

NeurIPS 2024

22 of 22

Thank you.