Parameter efficient LLM based vision-language models

Team 11: Omid Reza Heidari, Li Gu 2024-03-13

LLM-based vision-language models

What are Multi-Modal LLMs?

​

​

What is Key features of them?

• Integrated Understanding
• Generative Capabilities
• Adaptability

​

LLM based Vision-language Methods

Vision Encoder

Pre-trained LLM

Mapping Network

It is United States. I think so because the flag is the United States flag.

What country is this? Why do you think so?

Image: https://unsplash.com/photos/a-city-street-with-a-roller-coaster-in-the-background-oQ0MOFu7NOY

Why parameter efficiency?

There are some limitations with fine-tuning the entire model:

​

• Destroying the current capability of the LLM

​

• Training a huge number of parameters requires collecting a giant dataset

​

• Requires too much GPU memory as well as GPU hours

LLM based Vision-language Methods

Image: https://unsplash.com/photos/a-city-street-with-a-roller-coaster-in-the-background-oQ0MOFu7NOY

Vision Encoder

Pre-trained LLM

Mapping Network

It is United States. I think so because the flag is the United States flag.

What country is this? Why do you think so?

LLM based Vision-language Methods

Image: https://unsplash.com/photos/a-city-street-with-a-roller-coaster-in-the-background-oQ0MOFu7NOY

Vision Encoder

Pre-trained LLM

Mapping Network

It is United States. I think so because the flag is the United States flag.

What country is this? Why do you think so?

Adapter Layer

LLM based Vision-language Methods

Image: https://unsplash.com/photos/a-city-street-with-a-roller-coaster-in-the-background-oQ0MOFu7NOY

Vision Encoder

Pre-trained LLM

Mapping Network

It is United States. I think so because the flag is the United States flag.

What country is this? Why do you think so?

LLM based Vision-language Methods

Image: https://unsplash.com/photos/a-city-street-with-a-roller-coaster-in-the-background-oQ0MOFu7NOY

Vision Encoder

Pre-trained LLM

Mapping Network

It is United States. I think so because the flag is the United States flag.

What country is this? Why do you think so?

Drawback of other methods

• Number of parameters
• Overfitting
• Resources

MAPL - Architecture

A dog running in a beach.

Image: https://unsplash.com/photos/dog-running-on-beach-during-daytime-yihlaRCCvd4

Vision Encoder

Mapping Network

LM self-attention

LM Embedder

LM Tokenizer

A

running

a

.

dog

in

beach

A

running

dog

in

a

baech

.

MAPL - Mapping Network

Transformer Encoder

FC

FC

FC

FC

Learned Constant Embeddings

FC

FC

FC

FC

L_i Visual Features

​

Dimension: D_i

L_o Tokens

​

Dimension: D_o

L_o

Dimension: D_h

L_o Tokens, Dimension: D_h

MAPL - Mapping Network

FC

FC

FC

FC

FC

FC

FC

FC

• Decouple transformer hidden size D_h from D_i as well as D_o

​

• Shared Parameters in FC

MAPL - Training

After Training

Zero-shot transfer: captioning unseen images

few-shot transfer: unseen VQA

Training just mapping network from scratch by minimizing negative log-likelihood of the reference captions under the LM conditioned on the corresponding images.

MAPL - Experiments

Evaluation

Image Captioning

VQA (Not Actually)

Karpathy-test split of COCO

validation splits of

Conceptual Captions

TextCaps

VizWiz-Captions

Metrics

BLEU@4

ROUGE-L

METEOR

CIDEr

SPICE

validation splits of

VQAv2

OK-VQA

Text-VQA

VizWiz-VQA

Metric

VQA-Accuracy

Training Setting

domain-agnostic training

in-domain training

filtered version of CC(CC-clean) : 398k image-text pairs

Trained on 100% data

Trained on 1% data

MAPL - Image Captioning - Domain-agnostic

1%

100%

Existing Methods

MAPL - Image Captioning - Domain-agnostic

1%

100%

Existing Methods

MAPL - Image Captioning - in-domain

1%

100%

MAPL - Image Captioning - in-domain

1%

100%

MAPL - VQA - Domain-agnostic

​

1%

100%

Existing Methods

MAPL - VQA - Domain-agnostic

​

1%

100%

Existing Methods

MAPL - VQA - in-domain

​

1%

100%

MAPL - VQA - in-domain

​

1%

100%

Drawbacks of MAPL

There are several disadvantages that the MAPL model has, two of them can be addressed in the below:

• High GPU Usage: Even though we do not update the LM, we have to compute the gradient of it for the mapping network

• The LM should be open source (for computing gradient), and it goes without saying that open-source LMs are operated so poor in comparison to closed-source ones

BLIP-2: Bootstrapping Language-Image Pre-training

with Frozen Image Encoders and Large Language Models

​

ICML 2023

24

Overview

• Propose a parameter-efficient multimodal pre-training method that enables to bridge the vision-language modality gap
• Introduce a mapping network architecture (Adapter-free), named Q-former, along with a two-stage pre-training strategy
• Outperform Flamingo-80B by 8.7% on zero-shot VQAv2 but with 45x less trainable parameter.

Inference: Align the visual with the language

Key challenge: Since the LLM has not seen any images during its pre-training, how to “transform” the pure vision representation into a format (vision & language) that LLM can effectively use?

Align

Inference: Image Encoder + Q-former

Query transform (Q-former) = “filter”

• Extract language-informative visual representation from the frozen image encoder by conditioning the learned queries on the image feature

​

Query output

Language-informative

visual feature

Pure visual feature

​

​

Inference: Image Encoder + Q-former

Query transform (Q-former) = “filter”

• Extract language-informative visual representation from the frozen image encoder by conditioning the learned queries on the image feature

​

Query output

Language-informative

visual feature

Pure visual feature

​

​

Learned query

• Learnable parameters across the dataset
• A collection of language-related “filtering” criteria to guide Q-former

Inference: Image Encoder + Q-former

Query output

Pure visual feature

[HxW, D]

​

[N, D]

[N, D]

Query transform (Q-former)

• Extract language-informative visual representation from the frozen image encoder by conditioning the learned queries on the image feature
• Compress the resolution variant visual feature into a fixed number of query output

Language-informative

visual feature

Inference: Linear projection + LLM

Fully connected layer

• Linearly project the language-informative visual feature into the text embedding space

What is in the image?

• Serve as soft visual prompts for the LLM

Pre-training stage 1: Vision & Language representation learning

Q: How to learn a Q-former that can comprehend both visual and textual information to extract language-informative visual features?

Pre-training stage 1: Vision & Language representation learning

Q: How to learn a Q-former that can comprehend both visual and textual information to extract language-informative visual features?

​

​

​

​

A: Pretrain on three vision-language proxy tasks using image-text pairs

​

Pre-training stage 1: Vision & Language representation learning

​

Q: What is the architecture of Q-former? How to perform different proxy tasks?

• Two transformer submodules (image and text) that shares the same self-attention layers; Interaction between learned query and text
• Use cross-attention layers to condition the frozen image feature

Pre-training stage 1: Vision & Language representation learning

​

Q: What is the architecture of Q-former? How to perform different proxy tasks?

​

• For each proxy task, use different self-attention masks to control the interaction between queries and text.

Query

embed

Text

embed

Pre-training stage 1: Vision & Language representation learning

​

Q: What is the architecture of Q-former? How to perform different proxy tasks?

​

• For each proxy task, use different self-attention masks to control the interaction between queries and text.

Query

embed

Text

embed

Pre-training stage 1: Vision & Language representation learning

​

Q: What is the architecture of Q-former? How to perform different proxy tasks?

​

• For each proxy task, use different self-attention masks to control the interaction between queries and text.

Query

embed

Text

embed

Pre-training stage 1: Vision & Language representation learning

​

Q: What is the architecture of Q-former? How to perform different proxy tasks?

​

• For each proxy task, use different self-attention masks to control the interaction between queries and text for each proxy task
• For each image-text pair, combine three proxy losses into one total loss

Pre-training stage 2: Vision-to-Language generative learning

• Handle both encoder-decoder and decoder-only LLMs
• End-to-end training using language modeling loss, but only update Q-former and FC layer

A two-stage pre-training strategy

Text

Stage 1: Image-text pair loss

Stage 2: language generation loss

• Update Q-former at Stage 1; Update both Q-former and FC layer at Stage 2
• Both Flamingo and MAPL only have the pretraining on Stage 2

Pre-training Dataset & Benchmarks

Pre-training Data

• Total 129M images: COCO, Visual Genome, CC3M. CC12M, SBU and a subset of LAION400M; Instead, around 2B images used in Flamingo
• Create synthetic captions for the web images (still noisy)
• No a sequence of text interleaved with images and/or videos like Flamingo

Pre-training Dataset & Benchmarks

Pre-training Data

• Total 129M images: COCO, Visual Genome, CC3M. CC12M, SBU and a subset of LAION400M; Instead, around 2B images used in Flamingo
• Create synthetic captions for the web images (noisy)
• No a sequence of text interleaved with images and/or videos like Flamingo

​

Experiments

• Instructed Zero-shot Image-to-Text Generation: Zero-shot VQA
• Fine-tuned Image Captioning, VQA, image-text retrieval
• Fine-tuned for each specific downstream task
• Update both Q-former and ViT

Results: zero-shot VQA

• Outperform Flamingo-80B by 8.7% on VQAv2 with 45x fewer trainable parameters

Results: zero-shot VQA

• Underperform Flamingo-80B on Open-knowledge VQA due to a smaller LLM (BLIP2’s 11B FlanT5 vs. Flamingo’s 70B Chinchilla)

Results: Without Q-former pretraining

Demonstrate that decoupling the end-to-end training into two stages is crucial for state-of-the-art results.

Limitations

​

• Not perform in-context learning given few-shot examples
• No interleaved image-text pairs in the pretraining datasets

​

Limitations

​

• Not perform in-context learning given few-shot examples
• No interleaved image-text pairs in the pretraining datasets
• Only generate short sentences (average 6.5 words) that cover fewer objects
• Rely on more capable LLMs (e.g. Vicuna)
• Using noisy short image caption pairs is not sufficient.
• Instead, manually annotate more detailed image description datasets leading to natural language generation (e.g. mini-GPT4)

​

Limitations

​

• Not perform in-context learning given few-shot examples
• No interleaved image-text pairs in the pretraining datasets
• Only generate short sentences (average 6.5 words) that cover fewer objects
• Rely on more capable LLMs (e.g. Vicuna)
• Using noisy short image caption pairs is not sufficient.
• Instead, manually annotate more detailed image description datasets leading to natural language generation (e.g. mini-GPT4)
• Not perceive the local visual information in the image. E.g. Chart, Poster
• ViT-CLIP can only capture global information.
• The image feature in high resolution is compressed by Q-former into a small and fixed size (default 32) embedding
• Instead, infuse local visual features (e.g. object detection, segmentation) into Q-former

Summary

Parameter-efficient LLM-based vision-language models leverage LLMs’ strong capability (e.g. zero-shot, in-context learning) for vision-language tasks but with minimal changes to model’s architecture or parameters.

​

Two adapter-free approaches MAPL and BLIP2 reduce the need for high-scale trainable parameters, GPU resources, and pre-training datasets.

​

The next research directions may include:

• Train a mapping network in a black-box setting (only access LLM’s API)
• Enable models to comprehend local information in a high-resolution image

​

​

​

​

MAPL Vs FLAMINGO