Lecture 19: Multimodal Models Pre-training

Presenters: Eric Wang & Shengye Chen

Slides created for CS886 at UWaterloo

1

CS886: Recent Advances on Foundation Models

Overview

Slides created for CS886 at UWaterloo

2

• Introduction & Background
• Multimodal representations
• Multimodal tasks
• Visual Representation Improvement
• Oscar
• VinVL
• Contrastive Language-Image Pre-training
• CLIP
• ALIGN
• Generative Language-Image Pre-training
• BLIP
• CoCa
• Training Scaling Up
• SigLIP
• Review & Summarizations

Introduction & Background

Slides created for CS886 at UWaterloo

3

Multimodal Representations

Slides created for CS886 at UWaterloo

4

• Multimodality is the scientific study of heterogeneous (e.g., texts, images, audios) and interconnected data
• Goal: to represent and summarize multimodal data in joint representations s.t.
• complementary information is preserved
• redundant parts of the modalities are filtered out

​

​

​

​

​

​

• This presentation will only focus on Vision-Language Pre-training (VLP)

"Multimodal Machine Learning: A Survey and Taxonomy." https://arxiv.org/abs/1705.09406

Multimodal Tasks - Understanding

Slides created for CS886 at UWaterloo

5

• Image-text retrieval: predict whether the given image-text pair is aligned or not
• Image-to-text (I2T) retrieval: given an image, retrieve the best-fitting sentence from a pre-existing database
• Text-to-image (T2I) retrieval: vice versa
• Datasets: COCO, Flickr30k

• Natural Language Visual Reasoning (NLVR): given a pair of images and a text, determine whether the text statement is true about the images
• Datasets: NLVR, NLVR2

Multimodal Tasks - Understanding

Slides created for CS886 at UWaterloo

6

• Visual Question Answering (VQA): given an image and a question, select the correct answer from a multi-choice list
• Datasets:
• GQA: tests the reasoning capability of the model

Multimodal Tasks - Generation

Slides created for CS886 at UWaterloo

7

• Image captioning: generate a natural language description of the content of an image
• Datasets:
• COCO
• NoCaps: describe novel objects which are not in the training corpus

​

​

​

​

​

​

• Open-ended VQA

VLP Methods prior to 2020

Slides created for CS886 at UWaterloo

8

Existing methods (e.g., VisualBERT) simply concatenate visual region features and word embeddings of the paired text as input to the model and use self-attention to implicitly align elements of the text and image regions

"VisualBERT: A Simple and Performant Baseline for Vision and Language." https://arxiv.org/abs/1908.03557

VLP Methods prior to 2020

Slides created for CS886 at UWaterloo

9

Existing methods (e.g., VisualBERT) simply concatenate visual region features and word embeddings of the paired text as input to the model and use self-attention to implicitly align elements of the text and image regions

"VisualBERT: A Simple and Performant Baseline for Vision and Language." https://arxiv.org/abs/1908.03557

Visual Representation Improvement

Slides created for CS886 at UWaterloo

10

OSCAR: Object-Semantics Aligned Pre-training for Vision-Language Tasks

​

“Semantic alignments between texts and images using object tags”

March 2020

Slides created for CS886 at UWaterloo

11

OSCAR - Background & Motivation

Slides created for CS886 at UWaterloo

12

Motivation:

• Salient objects can be accurately detected by object detectors and are often mentioned in the paired text
• They can be used as anchor points for learning semantic alignments between image region features and word embeddings

Challenges:

• Ambiguity: Image region overlapping at different positions results in ambiguities for the extracted visual embeddings
• Lack of explicit alignments: There is no explicitly labeled alignments between regions or objects in image-text pairs

OSCAR - Extracting Anchor Points

Slides created for CS886 at UWaterloo

13

To extract visual embeddings v

1. Faster R-CNN is used to extract the visual semantics of each region as (v’, z)
1. v’: region feature, a vector of dimension P (e.g., 2048)
2. z: region position, a vector of dimension R (e.g., 4)
2. Concatenate v’ and z to form a position-sensitive region feature vector
3. Using a trainable linear projection to transform [v’, z] into v, a vector of dimension H (e.g., 768)

OSCAR - Extracting Word Embeddings

Slides created for CS886 at UWaterloo

14

To extract tag embeddings q

1. Faster R-CNN is used to extract the tags
2. Embed tags into word tokens q (H-dimensional) using pre-trained BERT

​

​

​

​

​

​

​

​

To extract text embeddings w

1. Embed tags into word tokens w (H-dimensional) using pre-trained BERT

Looking at the same input from 2 perspectives

15

Now that we have embeddings for texts (w), tags (q) and image regions (v), all in dim H

• Modality view:
• Text modality: word tokens (w)
• Image modality: image region features (v) & associated object tags (q)
• Goal: to distinguish the representations between a text and an image
• Dictionary view:
• Linguistic semantic space: word tokens (w) & object tags (q)
• Visual semantic space: image region features (v)
• Goal: to distinguish the semantic spaces between text and image

OSCAR - Loss for Modality View

Slides created for CS886 at UWaterloo

16

• Text modality representation w
• Image modality representation
• Pollute h’ s.t. it contains a set of images where the 50% tags are replaced with different tags
• Train a binary classifier f to predict whether image-text modality pair (h’, w) contains the original image or polluted ones
• Contrastive Loss
• Goal: to adjust word embedding space where a text is similar to its paired image and dissimilar to the polluted images

w

h’ = [q, v]

OSCAR - Loss for Dictionary View

Slides created for CS886 at UWaterloo

17

• Linguistic semantic space representation
• Visual semantic space representation v
• 15% tokens in h is replaced with [MASK] token
• Similar to masked language models, we want to predict masked text tokens (h_i) based on surrounding text tokens ( ) and all image features (v)
• Masked Token Loss
• Goal: to ground the learned word embeddings in the vision context

h = [w, q]

v

OSCAR - Pre-training

Slides created for CS886 at UWaterloo

18

• The total pre-training loss is
• Trainable parameters
• Linear projection matrix
• BERT
• Datasets:
• 6.5M image-text pairs consisting of 4M unique images
• COCO, Conceptual Captions (CC), SBU captions, Flickr30k, GQA

OSCAR - Quantitative Results

Slides created for CS886 at UWaterloo

19

Note that the dataset size of

• OSCAR: 6.5M image-text pairs
• Counterpart: over 9M image-text pairs

​

With fewer image-text pairs than SoTA_L, OSCAR_B achieves higher score than its counterpart in 5 out of 6 tasks, highlighting OSCAR’s parameter efficiency, partially because the use of object tags as anchor points eases the learning of semantic alignments between images and texts

OSCAR - The Effect of Object Tags

Slides created for CS886 at UWaterloo

20

• Training using predicted tags takes less than half of the training time to achieve the final performance of the baseline (no tags), showing the efficiency of utilizing object tags for VLP
• Training using ground truth tags further reduces the training time by over 50% to achieve the final performance of the predicted tags

OSCAR - Qualitative Results

Slides created for CS886 at UWaterloo

21

Intra-class: same object between two modalities is closer (e.g., person)

Inter-class: classes of related semantics are closer but still distinguishable, such as animal (zebra, elephant, sheep), transportation (train, car, truck), furniture (couch, chair, bowl).

OSCAR - Limitations

Slides created for CS886 at UWaterloo

22

• Requires a powerful object detector to handle complex scenes
• Does not work well when salient objects are missing in the text

A few good reasons to start with country line dance

VinVL: Revisiting Visual Representations in Vision-Language Models

​

"Extract better visual representation rather than just fuse multi-modal information"

Slides created for CS886 at UWaterloo

23

VinVL: Background & Motivation

Slides created for CS886 at UWaterloo

24

• Success of visual language pre-training (VLP) in visual language (VL) tasks
• VilBERT and OSCAR
• object detection (OD) model + cross-modal fusion model
• Vision-language fusion model
• OD model improvement untouched
• significance of visual features
• OD
• large-scale object-attribute detection model - ResNeXt-152 C4 (X152-C4)
• Visual Genome (VG) dataset

VinVL: Improve Vision in Vision Language

Slides created for CS886 at UWaterloo

25

• Idea:
• improve Vision for better visual representations
• enrich the visual object and attribute categories
• enlarge the model size
• train on much larger OD dasetset

• Mainstream
• Vision as a black box
• larger training dataset: OpenImages and Objects 365
• new insights in OD algorithms: feature pyramid network
• Powerful GPUs for bigger models

VinVL: Improve Vision in Vision Language

Slides created for CS886 at UWaterloo

26

• A new object detection model
• more accurate object-attribute detection results and better visual features for VL applications

Zhang, Pengchuan, et al. "Vinvl: Revisiting visual representations in vision-language models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

• X152-FPN: "boy"

• X152-C4: "young barefoot shirtless standing surfing smiling little playing looking blond boy"
• More than 20 additional object concepts

VinVL: Improve Vision in Vision Language

Slides created for CS886 at UWaterloo

27

2. Fine-tuning

• fine-tune the new OD model on VG to inject attribute information

1. Pre-training
• data
• 4 public complementary dataset - COCO, OpenImagesV5 (OI), Objects365V1, and Visual Genome (VG)
• build a unified corpus with VG vocabulary - sampling, balancing and merging
• model architecture: X152-C4
• model pre-training
• freezing: first conv layer, first res block and all batch-norm layers
• data augmentation: horizontal flipping and multi-scale training
• initialization from an ImageNet-5K checkpoint

VinVL: Revisit VL Models

Slides created for CS886 at UWaterloo

28

• Deep learning-based VL models:
• Vision: image understanding module
• VL: cross-modal understanding module
• Img: vision
• q: semantic representation of the image - object tag
• v: distributional representation of the image - visual representation
• w: language - text (question in VQA)
• y: output - text (answer to be predicted in VQA)

• Convention
• unify vision and language modeling VL with Transformer
• pre-train the unified VL with large-scale text-image corpora

VinVL: OSCAR+ Pre-training

Slides created for CS886 at UWaterloo

29

• Pre-training corpus
• three types of existing vision and VL dataset
• 8.85 million (w-q-Img) triples
• image captioning dataset
• visual QA dataset
• image tagging dataset

Pre-train an OSCAR+ to learn the joint image-text representations using image tags as anchors for image-text alignment.

OSCAR+ pre-training loss:

VinVL: OSCAR+ Pre-training

Slides created for CS886 at UWaterloo

30

OSCAR+ pre-training loss:

Masked Token Loss:

• defined on the text modality (w and q)
• define the discrete token sequence as
• apply the Masked Token Loss (MTL)
• randomly mask each input token with probability 15% and replace the masked one with a special token [MASK].
• predict the masked tokens based on their surrounding tokens and image features

VinVL: OSCAR+ Pre-training

Slides created for CS886 at UWaterloo

31

OSCAR+ pre-training loss:

Three-way Contrastive Loss:

• optimize the objectives for VQA and text-image matching
• training samples:

Negative examples for contrastive learning:

• polluted “captions”: (w′, q, v) for text-image matching task
• polluted “answers”: (w, q′, v) for VQA
• apply a FC layer on top as a 3-way classifier f( · ) given encoding of [CLS]
• triplet is matched (c = 0)
• triplet contains a polluted w (c = 1)
• triplet contains a polluted q (c = 2)

VinVL: OSCAR+ Pre-training

Slides created for CS886 at UWaterloo

32

• Pre-trained models
• language tokens = [w, q]
• region features = v
• BERT base and BERT large
• ensure that the features have the same input embedding size using a linear projection via matrix W
• trainable parameters are θ = {θBERT, W}

VinVL: Adapt to VL Tasks

Slides created for CS886 at UWaterloo

33

​

• Generation tasks - Image Captioning
• fine-tune
• training sample converted to a triplet: a set of captions, a set of image region features and a set of object tags
• seq2seq objective + uni-directional prediction with mask of 15% of the caption
• inference
• encode the image regions, object tags, and [CLS] as input
• generate a caption by feeding in a [MASK]

VinVL: Adapt to VL Tasks

Slides created for CS886 at UWaterloo

34

• Understanding tasks - VQA & GQA
• construct the input by concatenating a given question, object tags and object region features
• feed the [CLS] output from OSCAR+ to a task-specific linear classifier with a softmax layer

Contrastive Language-Image Pre-training

Slides created for CS886 at UWaterloo

35

Learning Transferable Visual Models From Natural Language Supervision (CLIP)

​

"Introduce self-supervised signals widely used in NLP into Vision"

Slides created for CS886 at UWaterloo

36

CLIP: Background & Motivation

Slides created for CS886 at UWaterloo

37

• Success of pre-trained models in NLP
• GPT family
• Zero-shot CV tasks
• 11.5% accuracy on ImageNet in 2017
• Improved performance in narrower and more targeted weak supervision
• SOTA CV systems
• Fixed set of predetermined object categories
• Low generality and usability
• CLIP-like methods
• VirTex, ICMLM, and ConVIRT: small scale training (< 1 million images)
• Close the gap
• Big data set: 400 million image-text pairs
• Large model size: ViT-large

CLIP: Contribution

Slides created for CS886 at UWaterloo

38

• Contrastive language-image pre-training

​

• Zero-shot beats task-specific supervised models

​

• Linear-probe with good performance

​

• Better generalization performance
• combine representation natural language and image

Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

CLIP: Method

Slides created for CS886 at UWaterloo

39

• Idea
• use natural language supervision signals to train a better visual model
• Why?
• no need to label data anymore
• images and text bound together to form a multi-modal feature

• Method
• create a sufficiently large dataset: Wikipedia-based Image Text (WIT) dataset - over 400 million image-text pairs
• select an efficient pre-training method
• choose and scale a model
• train

CLIP: Efficient Pre-Training Method

Slides created for CS886 at UWaterloo

40

• Solution
• contrastive learning
• Why?
• easy to only predict which text as a whole is paired with which image, instead of the exact words of the text

• Trial
• predictive task: training from scratch and predicting the caption of the image with CNN for image & Transformer for text
• Problem
• difficult and slow to predict the exact words corresponding each image

CLIP: Pre-training

Slides created for CS886 at UWaterloo

41

Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

• Contrastive pre-training
• contrastive learning on n × n features
• positive samples: image-text pairs on the diagonal
• negative samples: image-text pairs not on the diagonal

• Image encoder
• ResNet / Vision Transformer
• Text encoder
• Transformer

CLIP: Contrastive Training

Slides created for CS886 at UWaterloo

42

1

3

2

4

5

Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

CLIP: Contrastive Training

Slides created for CS886 at UWaterloo

43

Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

CLIP: Inference

Slides created for CS886 at UWaterloo

44

Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

• prompt template
• “A photo of a {label}, a type of pet”

• Inference without classification header
• cosine similarity

CLIP: Zero-Shot Classification Results

Slides created for CS886 at UWaterloo

45

Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

Across a 27 dataset eval suite, a zero-shot CLIP classifier outperforms a fully supervised linear classifier fitted on ResNet-50 features on 16 datasets, including ImageNet.

CLIP: Limitations

Slides created for CS886 at UWaterloo

46

• Computational efficiency
• SOTA performance on general dataset requires 1000x computation
• Weak zero-shot performance
• fine-grained classification
• abstract concepts: counting tasks
• new tasks un-existed in pre-training dataset
• Static classification
• non-generative model: image description
• Data efficiency
• 12.8 billion imagers in total requires 405 years with training one image / second
• Unrealistic zero-shot
• ImageNet

ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

​

"Scale of the corpus makes up for noise and leads to SoTA representations"

Slides created for CS886 at UWaterloo

47

ALIGN: Background & Motivation

Slides created for CS886 at UWaterloo

48

• Non-trivial data collection / cleaning in VL field
• CLIP
• Scaling of the corpus makes up for noise
• noisy dataset of over one billion image alt-text pairs: Conceptual Captions dataset
• An objective aligning the visual and language representations
• dual-encoder
• Image and text encoders learnt with contrastive loss
• a shared latent embedding space
• Aligned representations for cross-modality matching/retrieval tasks
• Zero-shot image classification

​

ALIGN: Noisy Image-Text Dataset

Slides created for CS886 at UWaterloo

49

• Trade quality for scale by relaxing most of the cleaning steps in the original work of Conceptual Captions dataset.
• Only apply minimal frequency-based filtering: aspect ratio, short dimension, content relevancy, text length, …

Scale up visual and vision-language representation learning.

Jia, Chao, et al. "Scaling up visual and vision-language representation learning with noisy text supervision." International conference on machine learning. PMLR, 2021.

ALIGN: Method

Slides created for CS886 at UWaterloo

50

Jia, Chao, et al. "Scaling up visual and vision-language representation learning with noisy text supervision." International conference on machine learning. PMLR, 2021.

Visual and language representations are jointly learned from noisy image alt-text data. The representations can be used for vision-only or vision-language task transfer. Without any fine-tuning, ALIGN powers zero-shot visual classification and cross-modal search including image-to-text search, text-to-image search and even search with joint image+text queries.

ALIGN: Pre-training on Noisy Data

Slides created for CS886 at UWaterloo

51

• Image encoder
• EfficientNet
• global pooling
• without training the 1x1 conv layer in the classification head
• Text encoder
• BERT with [CLS] token embedding
• A fully-connected layer on top
• Cosine-similarity combination function on top
• optimized via normalized softmax loss

• Training
• matched image-text pairs as positive and other as negative

Jia, Chao, et al. "Scaling up visual and vision-language representation learning with noisy text supervision." International conference on machine learning. PMLR, 2021.

ALIGN: Pre-training on Noisy Data

Slides created for CS886 at UWaterloo

52

Minimize the sum of two losses:

Image-to-text classification:

Text-to-image classification:

• Parameters
• xi and yj: normalized embedding of image in the i-th pair and that of text in the j-th pair respectively
• N: batch size
• σ: learnable temperature to scale the logits

ALIGN: Transferring

Slides created for CS886 at UWaterloo

53

• Image-text matching & retrieval
• w/wo fine-tuning
• dataset: Flickr30K, MSCOCO and CxC
• four intra- and inter-modal retrieval tasks
• three semantic similarity tasks
• Visual classification
• ALIGN zero-shot transfer
• dataset: same set (or a subset) of ImageNet classes
• Image encoder transfer
• dataset: ImageNet
• fine-grained classification dataset: Flowers-102, Oxford-IIIT Pet, Stanford Cars and Food101
• ImageNet
• training the top classification layer only with frozen ALIGN image encoder
• fully fine-tuned

ALIGN: Results

Slides created for CS886 at UWaterloo

54

Image-text retrieval

Jia, Chao, et al. "Scaling up visual and vision-language representation learning with noisy text supervision." International conference on machine learning. PMLR, 2021.

Zero-shot Visual Classification

ALIGN: Ablation Study

Slides created for CS886 at UWaterloo

55

A large scale training set is essential to allow scaling up of the models and to achieve better performance. A larger model is required to fully utilize the larger dataset.

Jia, Chao, et al. "Scaling up visual and vision-language representation learning with noisy text supervision." International conference on machine learning. PMLR, 2021.

Model quality improves nicely with larger backbones. As expected, scaling up image encoder capacity is more important for vision tasks. In image-text retrieval tasks the image and text encoder capacities are equally important.

ALIGN: Analysis of Learned Embeddings

Slides created for CS886 at UWaterloo

56

A simple image retrieval system to study the behaviors of embeddings trained by ALIGN.

​

ALIGN can align images and texts with similar semantics and generalize to novel complex concepts.

Jia, Chao, et al. "Scaling up visual and vision-language representation learning with noisy text supervision." International conference on machine learning. PMLR, 2021.

ALIGN: Analysis of Learned Embeddings

Slides created for CS886 at UWaterloo

57

ALIGN shows that word2vec-like linear relationships between word vectors emerge as a result of training them to predict adjacent words in sentences and paragraphs.

​

Given a query image and a text string, add their ALIGN embeddings together and use it to retrieve relevant images.

Jia, Chao, et al. "Scaling up visual and vision-language representation learning with noisy text supervision." International conference on machine learning. PMLR, 2021.

Generative Language-Image Pre-training

Slides created for CS886 at UWaterloo

58

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

​

“Improving text quality by bootstrapping contrastive training”

January 2022

Slides created for CS886 at UWaterloo

59

BLIP - Background & Motivation

Slides created for CS886 at UWaterloo

60

1. From model perspective: CLIP & ALIGN adopt encoder-based models

• Encoder-based models are not easily transferred directly to text generation tasks, such as image captioning
• Encoder-decoder models have not been successfully adopted for image-text retrieval tasks

To improve CLIP and ALIGN from 2 perspectives:

BLIP - Background & Motivation

Slides created for CS886 at UWaterloo

61

2. From data perspective:

• The number of high-quality human-annotated image-text pairs (e.g., COCO) is not enough for large multimodal model training
• CLIP & ALIGN are pre-trained on noisy web text, which can only yield suboptimal results

BLIP - Improving Caption Quality

Slides created for CS886 at UWaterloo

62

To solve the text quality issue, a natural approach is to build

• A discriminator to distinguish between good and bad image-text pairs
• A generator to synthesize better quality captions to replace noisy captions
• A unimodal encoder to align vision and language representations (similar to ALIGN)

Generator

(captioner)

Discriminator

(filter)

BLIP - Unimodal Encoder

Slides created for CS886 at UWaterloo

63

A multimodal alignment task to encourage matched image-text pairs to have similar representations in contrast to the negative pairs

​

Image-Text Contrastive (ITC) Loss

​

​

​

• x_i and y_j are normalized low-dimensional representations of [CLS] embeddings of text in the i-th pair and image in the j-th pair mapped by linear transformations
• Sum of 2 InfoNCE (Noise Contrastive Estimation) losses for I2T and T2I

BLIP - Discriminator (Filter)

Slides created for CS886 at UWaterloo

64

A binary classification task to predict whether an image-text pair is matched or nor, given multimodal features

​

Hard negative sampling strategy: negative pairs with higher contrastive similarity from ITC are more likely to be selected so that training is meaningful

​

Image-Text Matching (ITM) Loss

​

• is the predicted two-class probability
• is a 2-D one-hot vector representing the ground-truth
• H is cross-entropy loss

BLIP - Generator (Captioner)

Slides created for CS886 at UWaterloo

65

A generative task to produce textual descriptions in an autoregressive manner given an image

​

Language Modeling (LM) Loss

​

​

​

• y is the language tokens
• x is the image embedding

​

​

BLIP - Architecture

Slides created for CS886 at UWaterloo

66

The same color of blocks indicates shared parameters

BLIP - Bootstrapping Dataset with CapFilt

Slides created for CS886 at UWaterloo

67

​

1. Pre-train encoder & decoder with noisy web-scale dataset

​

​

​

​

2. Fine-tune filter and captioner using human annotated dataset (e.g., COCO)

​

​

​

​

3. Generate synthetic caption for web dataset.

Noisy text

Clean text

BLIP - Bootstrapping Dataset with CapFilt

Slides created for CS886 at UWaterloo

68

​

4. Filter synthetic and web captions to get high quality image-text pairs

​

​

​

​

5. Use high quality image-text pairs (129M, larger and cleaner) to pre-train a new model

​

​

​

​

​

Continue training does not help. This observation agrees with the common practice in knowledge distillation, where the student model cannot be initialized from the teacher

Noisy text

Clean text

BLIP - Bootstrapping Dataset with CapFilt

Slides created for CS886 at UWaterloo

69

These examples showcase the effectiveness of both captioner and filter

• Captioner is able to generate reasonable descriptions given an image
• Filter is able to accurately identify the more matched text

BLIP - Downstream Tasks

70

BLIP - Quantitative Results

71

Comparison between using captioner only and using filter only

• Captioner generates more diverse captions, which contain more new information that the model could benefit from

BLIP - Quantitative Results

72

Comparison between using CapFilt base and using CapFilt large

• Scaling up CapFilt from base to large only improves generative task performance
• Improvements of retrieval tasks is achieved by scaling up the vision backbone

BLIP - Quantitative Results

73

• The smallest BLIP outperforms ALIGN, despite using less than 1% of the data
• The smallest BLIP also outperforms ALBEF, which adopts encoder-based design and uses the same 14M images as BLIP without bootstrapping text

CoCa: Contrastive Captioners are Image-Text Foundation Models

​

“Combining contrastive training + generative training”

May 2022

Slides created for CS886 at UWaterloo

74

CoCa - Background & Motivation

75

• Recall: for each image-text pair, BLIP pre-training requires 1 forward pass through visual transformer and 3 forward passes through text transformers
• Need a minimalist design of BLIP to improve training efficiency

CoCa - Replacing Text Encoder with Decoder

Slides created for CS886 at UWaterloo

76

Append a [CLS] token at the end of input sentence and use its corresponding output of decoder as the text embedding

BLIP

CoCa

CoCa - Decoupled Decoder

Slides created for CS886 at UWaterloo

77

• Split the decoder into unimodal and multimodal components, by skipping the cross-attention mechanism in the unimodal decoder layers
• Benefit: the text decoder can efficiently generate outputs for both contrasive and generative losses with a single forward pass, compared to two passes for in BLIP

CoCa - Attentional Poolers

Slides created for CS886 at UWaterloo

78

Task-specific attentional pooling

• A multi-head attention layer with n_query learnable queries, with image encoder output as both keys and values
• n_query = 1 for ITC loss. Pooled image embedding as a global representation
• n_query = 256 for LM loss. More visual tokens are beneficial for region-level features

​

CoCa - Benefits of Attentional Poolers

Slides created for CS886 at UWaterloo

79

Adaptor for downstream tasks

• E.g., for video classification, a single query-token is learned to weight outputs of all tokens of spatial patches x temporal frames

​

Enhanced frozen-feature evaluation

• Linear probing struggles to accurately measure learned representations
• Learning a new pooler to aggregate features enables the model to obtain strong performance as a frozen encoder
• It can also benefit to multi-task problems that share the same frozen image encoder but different task-specific heads

CoCa - Pre-Training Details

Slides created for CS886 at UWaterloo

80

Loss Function

​

• λ are loss weighting hyper-parameters
• Empirically, a larger LM loss weight is better (λ_LM: λ_ITC = 2:1)
• Explanation: the ITC loss can be interpreted as a special case of the generative approach applied on image, when the vocabulary is the set of all captions

​

Number of unimodal and multimodal decoder layers

• N_{unimodal_decoder} = N_{multimodal_decoder}
• Intuitively, fewer unimodal text layers leads to worse zero-shot classification due to lack of capacity for good unimodal text understanding
• Fewer multimodal layers reduces the model’s power to reason over multimodal inputs such as VQA

​

Dataset (4.8B): ALIGN (1.8B) + JFT-3B (internal Google dataset)

CoCa - Evaluations

Slides created for CS886 at UWaterloo

81

CoCa outperforms foundation models and task-specialized models on 12 benchmarks including significant improvements in image-text retrieval, image captioning and VQA

Training Scaling Up

Slides created for CS886 at UWaterloo

82

SigLIP: Sigmoid Loss for Language Image Pre-Training

​

"Scaling up training with sigmoid loss"

Slides created for CS886 at UWaterloo

83

SigLIP: Background & Motivation

Slides created for CS886 at UWaterloo

84

• Contrastive pre-training
• weak supervision
• aligned representation space for images and texts
• CLIP and ALIGN
• contrastive objective
• Batch-level softmax-based contrastive loss
• pairwise similarity scores across all images, then all texts
• numerically unstable
• stabilization requiring additional pass over the full batch
• Sigmoid loss
• simplifying the distributed loss implementation
• symmetric sigmoid loss requiring just a single pass
• boosting efficiency
• decoupling batch size from definition of task

​

SigLIP: Softmax-based Contrastive Loss

Slides created for CS886 at UWaterloo

85

When using the softmax loss to formalize this objective, an image model f(·) and a text model g(·) are trained to minimize the following objective:

Due to the asymmetry of the softmax loss, the normalization is independently performed two times: across images and across texts.

, , scalar t is parametrized as exp(t′) , and

is a global freely learnable parameter.

Given a mini-batch B = {(I1, T1), (I2, T2), . . . } of image-text pairs.

SigLIP: Softmax-based Contrastive Loss

Slides created for CS886 at UWaterloo

86

Contrastive training typically utilizes data parallelism. Computing the loss when data is split across D devices necessitates gathering all embeddings with expensive all-gathers and the materialization of a memory-intensive |B| × |B| matrix of pairwise similarities.

​

Zhai, Xiaohua, et al. "Sigmoid loss for language image pre-training." arXiv preprint arXiv:2303.15343 (2023).

SigLIP: Sigmoid Loss

Slides created for CS886 at UWaterloo

87

Sigmoid loss does not require computing global normalization factors. It processes every image-text pair independently, effectively turning the learning problem into the standard binary classification on the dataset of all pair combinations, with a positive labels for the matching pairs (Ii , Ti ) and negative labels for all other pairs (Ii, Tj̸=i) . The loss is defined as:

zij is the label for a given image and text input, which equals 1 if they are paired and −1 otherwise.

​

An additional learnable bias term b similar to the temperature t is introduced to overcome heavy imbalance coming from the many negatives dominating the loss.

SigLIP: Sigmoid Loss

Slides created for CS886 at UWaterloo

88

L =

pair: - =

unpair: - =

SigLIP: Efficient Loss Implementation

Slides created for CS886 at UWaterloo

89

(a) (b)

Zhai, Xiaohua, et al. "Sigmoid loss for language image pre-training." arXiv preprint arXiv:2303.15343 (2023).

SigLIP: Efficient Loss Implementation

Slides created for CS886 at UWaterloo

90

(c) (d)

Zhai, Xiaohua, et al. "Sigmoid loss for language image pre-training." arXiv preprint arXiv:2303.15343 (2023).

SigLIP: Batch Size

Slides created for CS886 at UWaterloo

91

Apply sigmoid-based loss with CLIP and LiT:

​

Zhai, Xiaohua, et al. "Sigmoid loss for language image pre-training." arXiv preprint arXiv:2303.15343 (2023).

SigLiT results: Sigmoid loss outperforms the softmax loss significantly with small batch sizes, and performs similarly at larger batch sizes.

SigLIP results: Both sigmoid loss and softmax loss saturate at a reasonable batch size, while the peak of the sigmoid loss comes earlier and slightly outperforms the peak of the softmax loss.

SigLIP: Label Noise Robustness

Slides created for CS886 at UWaterloo

92

Sigmoid-training increases robustness to data noise.

​

Titles show the type of corruption applied, and x-axes show the probability with which they are applied. With increasing corruption severity, M-scale models trained with sigmoid loss for 3.6 billion examples retain superiority over corresponding softmax baseline.

​

Models trained with sigmoid loss are increasingly robust to all kinds of added noise.

Zhai, Xiaohua, et al. "Sigmoid loss for language image pre-training." arXiv preprint arXiv:2303.15343 (2023).

Summarizations

Slides created for CS886 at UWaterloo

93

Loss

Slides created for CS886 at UWaterloo

94

Contrastive loss

• ITC: sum of I2T and T2I InfoNCE loss to contrast paired text against others in the sampled batch (e.g., CLIP, BLIP)
• Sigmoid loss: binary classification of all pair combinations (SigLip)
• Binary classification to predict whether text-tag-image triplet contains the original tag or polluted tag (e.g., OSCAR)

Image-Text Matching (ITM) loss

• Binary classification to predict whether an image-text pair is matched or unmatched (e.g., BLIP)

Language Modeling (LM) loss

• A generative task to produce textual descriptions in an autoregressive manner given an image (e.g., BLIP)

Masked Language Modeling (MLM) loss

• Predict masked text tokens based on surrounding text tokens and image features (e.g., OSCAR, VinVL)

Uni-Encoder Family

Slides created for CS886 at UWaterloo

95

OSCAR

• Feeding the sequence of texts, tags and image regions embeddings to BERT
• Semantic alignments between texts and images using object tags
• Image-Text Contrastive (ITC) loss, Masked Language Modeling (MLM) loss

VinVL

• Improving OSCAR with a more powerful object detection model
• 3-way contrastive loss
• Same MLM loss as OSCAR

Dual-Encoder Family

Slides created for CS886 at UWaterloo

96

CLIP

• Introducing a learnable text encoder to encode free-form texts
• Image-Text Contrastive (ITC) loss

ALIGN

• Sacrificing quality to gain quantity – scaling up the corpus to 1.8B
• Extends dataset to multilingual to train ALIGNmling

SigLIP

• Changing softmax-based contrastive loss to sigmoid loss
• Advantages: memory efficient, fast, and numerically stable implementation

​

​

Encoder-Decoder Family

Slides created for CS886 at UWaterloo

97

BLIP

• Adding natural language generation capabilities
• ITC, LM, ITM loss
• Quality also matters – improving text quality by bootstrapping text

CoCa

• Minimalist design of BLIP, reducing the number of forward passes through transformer blocks
• ITC, LM loss
• Pre-trained with 4.8B images