Advancing Multimodal Vision-Language Learning

Aishwarya Agrawal

Assistant Professor @ UdeM and Mila

Research Scientist @ Google DeepMind

​

Multimodal AI Research (MAIR) Lab

Oscar Mañas

Saba Ahmadi

Le Zhang

Sarvjeet Singh Ghotra

Rabiul Awal

Kanishk Jain

Qian Yang

Shravan Nayak

Ankur Sikarwar

Rocktim Jyoti Das

(joining in Fall 2024)

Hanan Gani

(joining in Fall 2024)

(joining in Fall 2024)

Vision-Language Tasks

3

Vision-Language Tasks

4

Applications of vision-language systems?

• Aid to visually impaired users��

​

​

5

What kind of wine is this?

Photo and question are from vizwiz.org

Applications of vision-language systems?

• Aid to visually impaired users��
• Online shopping and organizing photos��

​

​

6

Applications of vision-language systems?

• Aid to visually impaired users��
• Online shopping and organizing photos��
• Grounded virtual assistants

​

7

Have you seen my keys?

Photo credits images.google.com

Vision-Language Progress

8

DeepMind’s Flamingo

Link

Vision-Language Progress

9

OpenAI’s DALL.E 2

Link

Vision-Language Challenges

• Visio-linguistic compositional reasoning

​

• Robust automatic evaluation

​

• Cultural and geo-diverse understanding

10

Vision-Language Challenges

• Visio-linguistic compositional reasoning

​

• Robust automatic evaluation

​

• Cultural and geo-diverse understanding

11

Visio-linguistic compositional Reasoning 

Which caption correctly matches the image?

Caption 1:

“The dog is on the left and the cat is on the right”

Caption 2 :

“The dog is on the right and the cat is on the left”

Visio-linguistic compositional Reasoning 

Which caption correctly matches the image?

Caption 1:

“The dog is on the left and the cat is on the right”

Caption 2 :

“The dog is on the right and the cat is on the left”

Contrastive VL Models Struggle

CLIP loss and pretraining data is too coarse to learn compositional relationship from

​

Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding

Our Method Overview

Our Method Overview

Our Method Overview

Hard Negative Caption Generation

“A black(adj) dog(noun) and a brown(adj) cat(noun) sitting(verb) on grass(noun) ”

Original Image-Text Pair

Augmented Hard Negative Captions

Swap Object (Relation)

Change Attribute (Attribute)

Change Verb (Action)

Change Object (Object)

Losses

Pull matched image-text pair closer and push unmatched pair away

Losses

Text

Encoder

Text

Encoder

text

similarity

(b) Intra-Modal Contrast

Push hard negative captions away from the corresponding positive captions

Losses

Text

Encoder

Text

Encoder

text

similarity

(b) Intra-Modal Contrast

Maintain a minimum similarity gap between positive image-text pair and hard negative image-text pair with an adaptive threshold

Adaptive Threshold

The threshold grows adaptively, providing stronger supervision signal as training progresses.

The CMR loss stabilizes after initial steps while the total loss keep decreasing

Results

We outperform previous state-of-the-art methods across several compositional benchmarks

Conclusions

• CLIP like models struggle with visio-linguistic compositional reasoning
• Automatically generating hard-negative captions and finetuning CLIP with them is promising
• Explicitly encouraging disparity between positive and hard-negative representations helps even further

VisMin: Visual Minimal-Change Understanding

26

Aishwarya Agrawal

Saba Ahmadi*

Rabiul Awal*

Le Zhang*

(*equal contributions, under submission at NeurIPS)

Data Generation Framework

Data Generation Framework

Data Generation Framework

75% of data filtered out

31

Spatial Relationship

Object

Attribute

Counting

A large scale synthetic data for model training (64K)

=>

object, attribute, counting and spatial relationship

​

Data Generation Framework

Additional 83% of data filtered out

33

Spatial Relationship

Object

Attribute

Counting

High quality human verified data for model benchmarking (2K)

=>

object, attribute, counting and spatial relationship

​

Dataset key features

• Visual Minimal Changes: Targeted change in one aspect: object, attribute, counting, or spatial relation.
• Visual Complexity: COCO and diffusion-generated images, edited by a detailed pipeline for targeted changes.
• Caption Complexity: Blend of human-written and LLM-generated captions.
• Human-Approved: Captions are sensical and images are natural looking.

Dataset distribution

Comparison of benchmarks offering visual hard negatives

• : Visual Minimal HN
• : Visual Complexity
• : Textual Complexity

​

• : Human-approved Captions
• : Human-approved Images
• : Size
• : Criterion holds for a subset of the benchmark

​

Benchmarking

The tasks involved two settings:

• Choosing the correct image from two captions and
• Selecting the correct caption from two images.

A man pouring white wine into a couple of wine glasses.

A man pouring red wine into a couple of wine glasses.

Caption 1

Caption 2

Image 1

Image 2

Image-text Matching Tasks

A man pouring white wine into a couple of wine glasses.

A man pouring red wine into a couple of wine glasses.

Caption 1

Caption 2

Image 1

Image 2

Image-text Matching Tasks

A man pouring white wine into a couple of wine glasses.

A man pouring red wine into a couple of wine glasses.

Caption 1

Caption 2

Image 1

Image 2

Text score

Sim(Image1, Caption1) > Sim(Image1, Caption2)

​

Image score

Sim(Image1, Caption1) > Sim(Image2, Caption1)

Foundation Models e.g. CLIP

Image-text Matching Tasks

A man pouring white wine into a couple of wine glasses.

A man pouring red wine into a couple of wine glasses.

Caption 1

Caption 2

Image 1

Image 2

Text score

Sim(Image1, Caption1) > Sim(Image1, Caption2)

​

Image score

Sim(Image1, Caption1) > Sim(Image2, Caption1)

Text score

Does this image best match: {Caption1 or Caption2}

​

Image score

Which image better aligns with the description: ‘{C}’? The first or the second image

Foundation Models e.g. CLIP

Multimodal large language model e.g. GPT-V

Benchmarking Foundational VLMs on VisMin

Object and attribute understanding in VLMs is very good, while attribute understanding is more challenging. Scaling model size also helps.

Benchmarking Foundational VLMs on VisMin

Models fail significantly on spatial relationships and counting categories. Increasing model size doesn’t help in understanding spatial relationships.

Benchmarking Multimodal LLMs on VisMin

Open-source MLLMs struggle more in image-score.

GPT4V has slightly better spatial relationship understanding, but still limited.

Finetuning VLMs on VisMin Training Set

Strong boost in performance for both foundational VLM and multimodal LLM.

Multimodal LLM benefits more from our training set, particularly for spatial relational understanding..

Finetuning VLMs on VisMin Training Set

We evaluate our finetuned model on multiple OOD benchmarks.�

Across the board we see VisMin training set brings notable gains → our training set improves fine-grained understanding in general

Finetuning VLMs on VisMin Training Set

Finetuning on our data improves performance on standard image-text retrieval for CLIP and

Recall results with ViT-L/14 on COCO

Benchmark

​

Evaluation standard VL tasks:MMMU and POPE

Finetuning VLMs on VisMin Training Set

Larger model has better gains from our training set.

Fine-tuning on CLIP variants. Circle radius reflects

the number of model parameters

Conclusions

• A new benchmark VisMin for fine-grained visual understanding evaluation.
• An automatic pipeline for generating visual minimal-change pairs.
• VLMs are good at object/attribute understanding, but poor at counting/spatial relation understanding.
• Finetuning VLMs on our visual minimal-change data helps!
• Improvement observed everywhere except CLIP on spatial relations
• Our dataset shows potential as a robust training resource for VLMs.

Webpage link: https://rabiul.me/vismin/

Learning to decompose complex questions into simpler sub-questions

50

(work in progress)

Aishwarya Agrawal

Qian Yang

Adaptive Decomposition for Complex Vision-Language Tasks

Qian Yang

Question: Think about the magnetic force between the magnets in each pair. Which of the following statements is true?

Options: A: The magnetic force is stronger in Pair 2.

B: The magnetic force is stronger in Pair 1

C: The strength of the magnetic force is the same in both pairs.

Label: B: The magnetic force is stronger in Pair 1

VLM (LLaVA): C: The strength of the magnetic force is the same in both pairs.

VLMs exhibit limitations in handling complex Vision-Language tasks.

Pair 1

Pair 2

30mm

16mm

Adaptive Decomposition for Complex Vision-Language Tasks

How to create an adaptive and reliable decomposition approach tailored to specific VLMs?

Qian Yang

• Overlooking VLM Feedback in Pre-Decomposition: Crucial feedback from VLMs is ignored, preventing adaptive adjustments based on VLM responses.

• Absence of Reliability Checks: Unreliable intermediate answers can bias the reasoning process, affecting the integrity of the conclusions.

• Previous Works focus on pre-decomposition.

Adaptive Decomposition for Complex Vision-Language Tasks

Qian Yang

• The Justifier assesses the confidence level of an answer and determines its reliability.

Question: Which of the following statements is true?

Options:

A: The magnetic force is stronger in Pair 2.

B: The magnetic force is stronger in Pair 1

C: The strength of the magnetic force is the same in both pairs.

VLM

C: The strength of the magnetic force is the same in both pairs.

Confidence: 45%.

Unreliable!

Justifier

• The VLM generates the initial answer.

Adaptive Decomposition for Complex Vision-Language Tasks

Qian Yang

• LLM generate the first sub-question based on the original question.

Question: Which of the following statements is true?

Options: A: The magnetic force is stronger in Pair 2. B: The magnetic force is stronger in Pair 1 C: The strength of the magnetic force is the same in both pairs.

Sub-Question 1: Are the magnets in each pair the same size and shape?

VLM

LLM

Sub-Answer 1: Yes

Confidence: 95%.

Reliable!

Justifier

• VLM answers the first sub-question conditioned on the image.

• Justifier validates the reliability of the sub-answer.

Adaptive Decomposition for Complex Vision-Language Tasks

Original Question: Which of the following …

Sub-Question 1: Are the magnets in each pair the same size and shape?

Sub-Answer 1: Yes. Confidence 95%

Qian Yang

• LLM adaptively generates subsequent sub-questions conditioned on the original question, all previous sub-QAs, and their reliability.

Sub-Question 2: Are the magnets in each pair attracted or repelled when placed close to each other?

VLM

LLM

Sub-Answer 2: Attracted.

Confidence: 93%

LLM

Sub-Question 3: Are the magnets in Pair 1 closer to each other than the magnets in Pair 2?

VLM

Sub-Question 1: Are the magnets in each pair the same size and shape?

Sub-Answer 1: Yes. Confidence 95%

• VLM answers each sub-question conditioned on the image and previous reliable sub-QAs.

Sub-Answer 3: Yes. Confidence: 89%

Adaptive Decomposition for Complex Vision-Language Tasks

Qian Yang

Sub-Question 1: Are the magnets in each pair the same size and shape?

Sub-Answer 1: Yes. Confidence 95%

Sub-Question 2: Are the magnets in each pair attracted or repelled when placed close to each other?

Sub-Answer 2: Attracted. Confidence 93%

Sub-Question 3: Are the magnets in Pair 1 closer to each other than the magnets in Pair 2?

Sub-Answer 3: Yes. Confidence: 89%

Question: Think about the magnetic force between the magnets in each pair. Which of the following statements is true? Options: A: The magnetic force is stronger in Pair 2. B: The magnetic force is stronger in Pair 1. C: The strength of the magnetic force is the same in both pairs.

VLM (LLaVA): B : The magnetic force is stronger in Pair 1.

Pair 1

Pair 2

30mm

16mm

Adaptive Decomposition for Complex Vision-Language Tasks

Qian Yang

• Performance Enhancement: Achieve a minimum of +4% improvement in SNLI-VE, ScienceQA, and A-OKVQA benchmarks.

Vision-Language Challenges

• Visio-linguistic compositional reasoning

​

• Robust automatic evaluation

​

• Cultural and geo-diverse understanding

58

Improving

Automatic VQA Evaluation

Using Large Language Models

VQA Accuracy

• VQA is traditionally evaluated with VQA Accuracy
• Based on exact string match (EM)

VQA Accuracy failure modes

34.25%

Multiple answers

​

​

Q: “What are the sticks for?”

A: “balance”, “pushing”, “skating”

27.75%

Over- or under- specifying and verbosity

​

Q: “Where is the hydrant?”

A: “on the right”, “right”

21.0%

Synonym

​

​

​

Q: “What is the setting of this picture?”

A: “field”, “plains”, “grassland”

18.0%

Broad/bad question or generic response

​

​

Q: “How many sheep are there?”

A: “many”

LLM-Assisted VQA Evaluation (LAVE)

+

+

Correlation with human judgment

Qualitative examples

An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics

65

Aishwarya

Agrawal

Saba

Ahmadi

What is the image captioning task?

66

Model

Input:

A boy smiling while jumping on a skateboard.

Output:

Example from Microsoft COCO: Common Objects in Context (Tsung-Yi et al.)

​

Robust Automatic Evaluation

• Evaluating image captioning is difficult!

[Hessel et al. EMNLP 2021]

• Recently, reference-free metrics have been proposed – CLIPScore, UMIC.

• Existing metrics rely on n-gram matches between candidate and reference captions.

Reference-free metrics

68

A boy smiling while jumping on a skateboard.

Similarity score

Robust Automatic Evaluation

Saba Ahmadi

Recently proposed reference-free image-captioning metrics are not robust enough!

• They fail to recognize fine-grained differences between correct and incorrect captions.

Figure credits: Saba Ahmadi

• They have poor understanding of negation.

• They are biased by the length of the captions.

Robust Automatic Evaluation

Saba Ahmadi

Recently proposed reference-free image-captioning metrics are not robust enough!

• CLIPScore is more sensitive (than UMIC) to the number and size of objects mentioned in the caption.

Figure credits: Saba Ahmadi

Robust Automatic Evaluation

Saba Ahmadi

Recently proposed reference-free image-captioning metrics are not robust enough!

• CLIPScore is more sensitive (than UMIC) to the number and size of objects mentioned in the caption.

Figure credits: Saba Ahmadi

• CLIPScore is indifferent to the sentence structure.

Robust Automatic Evaluation

Saba Ahmadi

Recently proposed reference-free image-captioning metrics are not robust enough!

• CLIPScore is more sensitive (than UMIC) to the number and size of objects mentioned in the caption.

Figure credits: Saba Ahmadi

• CLIPScore is indifferent to the sentence structure.

Vision-Language Challenges

• Visio-linguistic compositional reasoning

​

• Robust automatic evaluation

​

• Cultural and geo-diverse understanding

73

Motivation: Western centric bias

74

Web

COCO, LAION, CC-12M, WIT, CoYo

Motivation: Western centric bias

ImageNET and COCO primarily consist of images from North America and Europe. Many benchmarks built upon these inherit the same biases.

Source: Does Object Recognition Work for Everyone? De Vries et.al. 2019

Benchmarking geo-diverse cultural understanding in VLMs

Shravan Nayak

Kanishk Jain

Rabiul Awal

Karolina Stańczak

Lisa Anne Hendricks

Sjoerd van Steenkiste

Siva Reddy

Aishwarya Agrawal

Culture from VL perspective

Culture is a multifaceted concept that may refer to but is not limited to beliefs, practices, symbols, and norms found in human society.

Vision-Language perspective

Visible Cultural Manifestations

Invisible Cultural Manifestations

Cross-Cultural Analysis: The Science and Art of Comparing the World's Modern Societies and Their Cultures. Michael Minkov.

Visible cultural manifestations

Food

Clothing

Drinks

Rituals

Cross-Cultural Analysis: The Science and Art of Comparing the World's Modern Societies and Their Cultures. Michael Minkov.

Invisible cultural manifestations

Societal norms

Legal frameworks

Beliefs

Values

Cross-Cultural Analysis: The Science and Art of Comparing the World's Modern Societies and Their Cultures. Michael Minkov.

Invisible cultural manifestations can't be seen directly, but their impact is often visible in people's actions and behaviors.

From a VL perspective, culture means not only identifying and describing visual concepts but also understanding the underlying values and socio-cultural contexts they reflect

Values

Norms

Beliefs

Image Source Left to Right: Trip Savy, India Times, Getty Images

CANDLE Dataset

81

Size:

60k clusters

1.1 M raw sentences

Domains:

• Geography
• Religion
• Occupations
• States of U.S.A

Nguyen et.al. Extracting Cultural Commonsense Knowledge at Scale

Built by filtering C4

Consists of Cultural Commonsense Knowledge (CCSK) assertions.

Pairing assertions with images from web

82

Cultural Repository of Images and Text (CRIT)

People in Brazil wear either black or purple to represent grief and loss.

The festival features Nigerian 'voodoo spirits' walking the streets.

Cultural Repository of Images and Text (CRIT)

Pav Bhaji is an Indian fast food dish

The lion dance is an important tradition in Chinese culture

Cultural Repository of Images and Text (CRIT)

Janmashtami is known and celebrated by different names across in India

Mexican Horchata is a refreshing and slightly sweetened rice milk drink

Cultural Repository of Images and Text

CulturalVQA

• A systematic way to evaluate understanding of cultural nuances in VLMs → Visual Question Answering
• Solicit question and answer annotations from participants familiar with the culture
• Choose annotators from different cultures/countries

Culturally knowledgeable human annotators

CRIT Images +

Metadata

CulturalVQA

Collecting Questions

We ran several studies on MTurk to collect culturally nuanced questions for images.

Additional conditions that questions must satisfy

• The question must require an understanding of your culture to to answer it correctly
• The question must require looking at the image to answer it correctly
• The question must elicit a single correct answer.
• Do not ask a question based on stereotypes i.e., oversimplified beliefs about your cultural group

Your task is to ask a question about the cultural concept depicted in the image that someone from your culture will be able to answer easily, but someone who is not familiar with your culture will not be able to answer.

Collecting Answers

1. Be culturally precise: Your answer should be the precise term people from your culture would use. Your answer should not be generic.

“Holiday” -> “Christmas”�

• Use English: Your answer should be written in English. Use regional terms only if there is no direct English translation for that term.� “Dhaniya patta” -> “Coriander leaves”� “Naan” -> “Naan” (instead of “Indian bread”)

Culturally precise

Indian Bread

​

Naan

What food item is shown in the image?

Dhaniya Patta

​

Coriander leaves

What is the green substance in the image?

Challenges

• Difficulty in sourcing annotators
• Most annotators from US and India. African countries have little to no annotators available
• We had to pool annotators from multiples sources: MTurk, Masakhane (African regions), Universities in Montreal
• Adhering to instructions
• Language barrier: Annotators from non-English speaking countries have a hard time referring to images in questions.

CulturalVQA Benchmark

Our efforts are focused on collecting data from the following countries

Examples

What does the bride-to-be add to the groom-to-be's drink in the image, when he asks for her family's blessing to marry?

​

Ans: Salt

Turkey

What is around the bride and groom's neck?

​

Ans: Varmala

India

Examples

Canada

What do the feathers on his head mean?

​

Ans: Chief

Nigeria

The beaded headgear is from which culture?

​

Ans: Edo culture

CulturalVQA: Statistics

CulturalVQA: Statistics

CulturalVQA: Statistics

Evaluation Metric

Leverage LAVE metric which utilizes in-context learning capabilities of instruction tuned LLMs for VQA evaluation by formulating it as an answer-rating task

Improving Automatic VQA Evaluation Using Large Language Models; Manas et. al.

Evaluation Metric

When is the object shown in the image usually used?

Ground Truth: praying or religious offerings

​

Model: Prayer time

​

String Matching:

​

LAVE:

Evaluation Metric

What does the food kept in baskets represent in the image?

Ground Truth: Prasadam

​

Model: Prasad

​

String Matching:

​

LAVE:

Benchmarking VLMs on CulturalVQA

Models perform better for North America than for Africa / Asia.���Closed source models are strictly better than open source models���Gap between closed-source and open-source increases as we go beyond North America!

Benchmarking SOTA VLMs on CulturalVQA

​

GPT-4 outperforms all models with Gemini being second best.

�Intern VL 1.6 is the best open source model (larger size plus more data)

​

Question: What type of Indian bread is this in the picture?

​

Pred: Khakhra

GT: Naan

Question: What is the local name for the white-colored food in the image in East Africa?

​

Pred: Ugali

GT: Kawunga

Question: For what occasion is the flower typically displayed in front of the capital building?

​

Pred: Tulip Festival

GT: Remembrance Day

Question: The beaded headgear is from which culture?

​

Pred: Benin Kingdom

GT: Edo culture

GPT4 Failure Cases

Conclusions

• A large scale repository of culturally relevant image-text pairs.�
• CulturalVQA – a new task to evaluate VLMs’ geo-diverse cultural understanding. �
• Comprehensive evaluation of both open-source and closed-source VLMs on the CulturalVQA benchmark showing interesting findings.

Vision-Language Challenges

• Visio-linguistic compositional reasoning
• VLMs are still poor at visio-lingistuic compositional reasoning
• Fine-tuning with high-quality visual and textual hard-negatives is promising
• Explicitly encouraging disparity between positive and hard-negative representations helps even further

106

Vision-Language Challenges

• Visio-linguistic compositional reasoning

​

• Robust automatic evaluation
• VQA: Using LLMs for reference-based evaluation is promising!
• Image Captioning: Reference-free evaluation is not robust enough!

107

Vision-Language Challenges

• Visio-linguistic compositional reasoning

​

• Robust automatic evaluation

​

• Cultural and geo-diverse understanding
• Need to go beyond traditional recognition problems now that VLMs are quite strong
• Evaluating cultural understanding is not easy
• Although current VLMs have decent cultural understanding for North America, they perform poorly for Africa and Asia.

108

Vision-Language Challenges

• Visio-linguistic compositional reasoning

​

• Robust automatic evaluation

​

• Cultural and geo-diverse understanding

109

Thanks!�Questions?