DAReN�A Collaborative Approach Towards Visual Reasoning And Disentangling

Pritish Sahu, Kalliopi Basioti, Vladimir Pavlovic

Rutgers University

2022 Conference on International Conference of Pattern Recognition (ICPR 2022)

Problem Statement

Raven’s Progressive Matrices (RPM)

Row 1

Shapes, scales are different. Color remains constant.

Shapes, scales are different. Color remains constant.

Row 2

2

Problem Statement

Correct!

All colors blue.

Raven’s Progressive Matrices (RPM)

Row 1

Shapes, scales are different.

Row 2

Shapes, scales are different.

Color remains constant.

Row 3

Wrong!

Colors are different.

Correct Answer

3

Abstract Structure Of RPM

Constant in a row*

Quantitative pairwise progression

Figure addition/subtraction

Distribution of three values Distribution of two values

* For our experiments, we consider the “constant-in-a-row” relationship.

Object Type

Object Size

Position (XY–axis)

Color

For e.g.,

4

Inferring Factors from Visual Signal

Factor: Nuisance

Change in one latent factor results in variation of only one aspect in the image, exclusively.

5

Latent Factors

Reasoning + Disentangling

R

w

1

R

o

w

2

1. RPM

Relation = Constant in a row

(c) Reasoning

Choice List

Color

​

Shape

​

Scale

6

Prior Work

“Are Disentangled Representations Helpful for Abstract Visual Reasoning?”^[1]

Stage 1

Unsupervised Representation Learning

Stage 2

Reasoning (Wild Relational Network)^[2]

7

g

Choice Scores

Graphical Model of RPM (GM-RPM)

Graphical Model-RPM

Row 1

R

Row 2

Row 3

G

B

8

DAReN

Generative Model

Context

Matrix

Choice

List

Latent Representation of images  in the RPM

RPM​

9

DAReN

Reconstructed

images of solved RPM puzzle

10

DAReN

Reasoning Model

Training Objective

11

Datasets and Evaluation Metric

dSprites

Modified dSprites

Shapes3d

MPI3D

• Adapt the above datasets to generate raven’s progressive matrices (RPM)*

Context

Matrix

Choice

List

* Adapted from “Are Disentangled Representations Helpful for Abstract Visual Reasoning?”^[1]

12

Datasets and Evaluation Metric

• Reasoning: accuracy of reasoning model on unseen RPM questions.

​

• Disentanglement Metrics:
• FactorVAE Score (FVAE): Accuracy of majority vote classifier that predicts the index of a fixed factor.
• DCI Disentanglement Score (DCI): Entropy of the latent / factor predictive importance over factors.
• Mutual Information Gap (MIG): Normalized gap in latent / factor MI between top two latents.
• Separated Attribute Predictability (SAP): Avg. difference in latent / factor prediction error between top two latents.

​

​

13

Quantitative Results

• We train over 35 models with different hyper parameter settings.

1. E2E-WReN: End-to-end learning of disentanglement and reasoning, which is our adaptation of Staged-WReN (prior work).

2. In the table above, color red means best performance and blue is the second-best performance.

Shapes3D

Modified Dsprites

14

Qualitative Results – Latent Traversals

• Latent traversal with relevance indicators

Modified dSprites

Shapes3D

-3

3

KL

Divergence

-3

3

0

0

Relevance

​

Nuisance

15

Conclusions

• New graphical model for RPM (GM-RPM).
• Established the dependency of disentangled latent factors and visual reasoning.
• DAReN exploits the weak inductive bias in RPM to jointly learn reasoning and disentangled representation.
• DAReN
• State-of-the-art results on both the reasoning and disentanglement tasks for all benchmark datasets.
• Offers flexibility of using any SOTA factor inference approaches based on ELBO-like objectives.
• Results point to strong correlation between learning disentangled representation and solving the reasoning tasks.

16

References

1. Van Steenkiste, Sjoerd, et al. "Are disentangled representations helpful for abstract visual reasoning?." Advances in Neural Information Processing Systems 32 (2019).

2. Barrett, David, et al. "Measuring abstract reasoning in neural networks." International conference on machine learning. PMLR, 2018.

17

Thank you