Learning Transferable Visual Models From Natural Language Supervision

​

​

By Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger and Ilya Sutskever

Presented by Balaji Balasubramanian and Eshwanth Baskaran

Introduction

• Labelling Data for supervised learning tasks
• Expensive
• Self-Supervised learning
• Learning from input data itself without labels.
• Pretext task (pre-defined task) is defined for the model to solve which helps model to learn useful features.
• The features learned can be used for downstream tasks.

L. Jing and Y. Tian, ‘‘Self-supervised visual feature learning with deep neural networks: A survey,’’

CLIP - What is it?

• Connecting text and images…

​

​

​

​

​

• Formally…
• Method to learn task-agnostic image representations using natural language supervision.
• CLIP - Contrastive Language-Image Pretraining

CLIP - Motivation

• Success of language models like GPT-3…
• Uses web-scale raw, text data
• Zero-shot transfer to downstream tasks.
• CLIP…
• aimed at developing GPT-3 like models for vision classification problems.
• Once trained…
• Can be applied to any visual classification benchmark (zero-shot).

CLIP - Dataset curation

• Benchmark image-text pair datasets:

Only around 100,000 images

​

Noisy and sparse metadata - filtering reduces data by 6x

MS-COCO

YFCC100M

CLIP - Dataset curation

​

• Manual curation - image-text pairs largely available publicly on internet.
• 400 million image-text pairs crawled from internet.
• Final dataset - WebImageText (WIT)
• 400 million image-text pairs
• Proprietary

CLIP - Problem formulation

Generative model

Predict

Input

Features

Features

Text encoder

Image encoder

Contrastive loss

Input

Input

Predict exact words…

HARD TASK

MORE COMPUTE

Predict which text pairs with which image

EASIER TASK

LESS COMPUTE

EFFECTIVE

VS

Tian, Y., Krishnan, D., and Isola, P. Contrastive

multiview coding. arXiv preprint arXiv:1906.05849, 2019.

CLIP - Architecture

• Text encoder
• Encodes natural language text into multi-modal space

​

• Image encoder
• Encodes image features into the same multi-modal space

CLIP - Models

• Visual encoder:
• 5 ResNet Model variants
• ResNet-50
• ResNet-101
• ResNet-50x4, ResNet-50x16 and ResNet-50x64
• Scaled EfficiencyNet-style models
• 3 Vision-Transformer (ViT) variants
• ViT-B/32 (Base model)
• ViT-B/16
• ViT-L/14 (Large model) - BEST PERFORMANCE
• Text encoder
• GPT-2 like transformer

Visual Transformer (ViT) - Architecture

CLIP vision transformers are about 3x more compute efficient than CLIP ResNets

Text Encoder - GPT-2

• Lower-cased BPE for token embedding
• 49,152 - vocab size
• 63M parameter model
• 12 layers
• 512 dimensions wide
• 8 attention heads

CLIP - Training

1

2

3

4

• Mini-batch size - n
• Text feature dim - d_t
• Image feature dim - d_i
• Multi-modal space dim - d_e

Training - Temperature parameter

• Knowledge distillation concept
• Softmax with temperature parameter (soft-targets)

https://openaccess.thecvf.com/content_cvpr_2018/CameraReady/0801.pdf

CLIP - Training configuration

• Input image - 224x224 dims
• Tried 336x336 dims for ViT-L/14 - BEST PERFORMANCE
• Optimizer - Adam with weight decay regularization
• Minibatch size - 32,768 image-text pairs
• Epochs - 32

Want the minibatch size to be larger, but hardware constraints

1. With enough compute and data, models can learn effective representations (ViT)
2. Negate pretrained bias

CLIP - Test phase (zero-shot)

Test image

Test categories

Prediction

CLIP - Comparison with classification models

1. Better representations
2. Works on diverse tasks
3. No manual labelling
4. Using language supervision helps understand context (verbs)
• Avoids polysemy - classify better
• Helps in action recognition dataset

CLIP

Classifier

Input

CLIP as a linear classifier

• The cosine similarity between the text and image representations is given by
• y = softmax(T⋅I) where T is the text feature matrix and I is the image feature matrix
• The above equation is similar to y = softmax(W^TX) - a linear classifier with no bias.
• Hence, we can perceive the CLIP model as a linear classifier with:
• image encoder (visual backbone) feature as input (X)
• Text encoder feature as the corresponding weight for the given input image.

Related works- Virtex

• Jointly trains an image CNN and text transformer using image caption pairs for the task of image captioning.
• Learned CNN is used for downstream tasks.

Desai et al.- VirTex: Learning Visual Representations from Textual Annotations

Related works- ConVIRT

• Trained on pairs of Chest Radiograph images and text description of characteristics/abnormalities.
• The model maximizes the agreement between the true image-text representation pairs with bidirectional losses

Yuhao Zhang et al- Contrastive Learning of Medical Visual Representations from Paired Images and Text

Related works- ICMLM

• Image-conditioned masked language modeling (ICMLM)
• The model learns visual representations from image-caption pairs
• Masks tokens in captions are predicted by fusing visual and textual cues.
• Visual attention changes as different tokens in a caption are masked.

Mert Bulent Sariyildiz et al- Learning Visual Representations with Caption Annotations

Experiments

Prompt Engineering and Ensembling

• Customizing prompt text to each task improves zero shot performance.
• Use ‘A photo of a dog, a type of pet.’ to improve performance further.
• Ensemble models using different prompts i.e. ensembling over embedding space instead of probabilistic space.
• This is an image of a boxer, a breed of dog. Boxer can be a type of athlete or a breed of dog, providing context to the text prompt helps improve the model performance.

​

Experiments

Zero Shot CLIP vs Linear Probe on Resnet 50

• Linear probe on Resnet 50 means that a fully supervised linear classifier is fitted on ResNet-50 features.
• Clip outperforms Resnet in 16/27 datasets.
• Clip performs better for Cars and pets and Resnet performs better for Birds and Aircraft.
• Probably due to varying amount of per-task supervision between WIT vs Imagenet.

Experiments

Zero Shot CLIP vs Linear Probe on Resnet 50

• On general datasets like CIFAR10, STL10, ImageNet CLIP performs slightly better due to a larger and more varied training dataset for CLIP.
• On action recognition datasets like Kinetics700 and UCF101, CLIP outperform Resnet because natural language provides better supervision for verbs.

Experiments

Zero Shot CLIP vs Linear Probe on Resnet 50

• On abstract tasks such as counting objects in synthetic scenes(CLEVRCounts), Satellite Image Classification(EuroSAT), CLIP performs poorly.
• Surprisingly, it performs poorly on MNIST due to lack of overlap between WIT and MNIST datasets.

Question

• As we have seen in the previous few slides, CLIP performs well for general datasets, action recognition.
• It performed well for few specific classes like birds and pets but it performed badly for classes like birds and aeroplanes depending on the amount of supervision it received during training.
• But it performed badly for abstract tasks like counting, and certain image types that it did not encounter during the training like Satellite Images and MNIST.
• CLIP mainly increases the distribution by using a large dataset from the internet and generating free labels using text captions. This model does not address the problem of out of distribution generalization.

Question

• Florence can be used for more complex tasks like object detection, vqa and complex categories like satellite imagery, chest x-ray images.
• It uses more better architectures like Swin Transformer, and performs better model finetuning than linear probe.

Yuan et al.- Florence: A New Foundation Model for Computer Vision

Liu et al.- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Experiments

CLIP with Linear Probe

• Zero Shot CLIP performs as well as 4 Shot CLIP with linear probe because the number of samples are too few to effectively train the linear classifier.
• The performance of linear probe CLIP increases with the increase in the number of samples per class.
• Training linear probe with all samples is the upper bound for CLIP.

Experiments

CLIP with Linear Probe

• Training Linear Probe with CLIP improve its performance by 10-25% for most datasets.

Experiments

CLIP with Linear Probe

• Train Logistic Regression on CLIP performs better than Noisy Student EfficientNet L2 on 21/27 datasets
• EfficientNet outperforms CLIP on ImageNet dataset because it was trained on it.

Xie et al. - Self-training with Noisy Student improves ImageNet classification

Experiments

Compute NSL for CLIP

• CLIP’s performance improves smoothly with the increase in compute.
• The average zero shot loss across 36 dataset decreases consistently.
• The individual loss for the datasets are much noisier.

Experiments

Compute NSL for CLIP

• VIT CLIP performs better than Resnet CLIP.
• CLIP VIT are 3x more compute efficient than CLIP Resnet.

Experiments

• Zero-shot CLIP is much more robust to distribution shift than standard ImageNet models. ViT-L/14 vs Resnet-101
• The robustness is due to large dataset size (not specific to imagenet classes) , natural language supervision (word synonyms).

Experiments

• The model is adapted to Imagenet using Linear Probe.
• While supervised adaptation to ImageNet increases ImageNet accuracy by 9.2%, it slightly reduces average robustness.

​

Broader Impacts

• CLIP has a wide range of capabilities and can be tuned to a wide range of application using zero-shot learning and linear probe.
• But it also has the potential to amplify the inherent biases in data.

Bias

• CLIP has been test on FairFace dataset.
• FairFace is a face image dataset designed to balance age, gender, and race.
• It categorizes gender into 2 groups: female and male and race into 7 groups: White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latino.

K. Kärkkäinen and J. Joo, “FairFace: Face attribute dataset for

balanced race, gender, and age,” Aug. 2019.

Bias

• FairFace Model - Resnet34
• Instagram Model- Transfer learning of ResNeXt-101 to predict hashtags on billions of social media hashtags.
• Zero Shot CLIP performs better than FairFace model on 3/6 categories.
• Linear probe CLIP performs better than FairFace models on most categories.

K. Kärkkäinen and J. Joo, “FairFace: Face attribute dataset for

balanced race, gender, and age,” Aug. 2019.

Mahajan et al. : Exploring the Limits of Weakly Supervised Pretraining

Race, Age, Gender Classification for Non-White Categories

Applications- Image Search

• Get Image Features by passing the images through the CLIP model and storing it.
• Compute Similarity between the image features and text features.

https://mikhalevi.ch/rclip-an-ai-powered-command-line-photo-search-tool/

Applications- Image Similarity

• Image Similarity- These images match even though they have different watermarks.

https://blog.roboflow.com/openai-clip/

Applications- Deciphering Corrupted Images

• CLIP performs better than Imagenet trained Resnet-101 for corrupted and adversarial images.

Sriram et al. - Inverse Problems Leveraging Pre-trained Contrastive Representations

CLIP - How is it better?

1. CLIP is highly efficient
• Earlier: ResNeXt101-32x48d - 19 GPU years for Imagenet classes
• Training efficiency
1. Contrastive objective ~ 4x less compute
2. Vision Transformer ~ 3x less compute
• Learning visual concepts directly from natural language
• Flexible and general than existing ImageNet models.
2. CLIP is flexible and general
• Diverse tasks
• Robust to distribution and domain

1. Fine-grained object classification
2. Action recognition in videos,
3. OCR
4. ……..

​

CLIP - Limitations

1. Doesnt work well on abstract or complex tasks
• Counting the number of objects in an image, etc
2. Zero-shot CLIP struggles on fine-grained classification tasks
• Tasks to differentiate between bird categories, car models, etc

CLIP - Limitations

​

3. Poor generalization to images not covered in its pre-training dataset.

• Zero-shot CLIP only achieves 88% accuracy on MNIST dataset.

4. Zero-shot classification - sensitive to wording or phrasing

• Trial and error “prompt engineering”

CLIP - Limitations

​

5. Typographic attacks

• Multi-modal neurons - learns concepts

​

https://arxiv.org/pdf/2103.10480.pdf

Conclusion

• CLIP is one of the first foundation models for images.
• Zero-shot not meant for commercial deployments.
• Not perfect but…
• Good starting point for researchers to understand zero-shot image classification tasks.