CSE641 - Deep Learning Winter 2022

Group 35

Lightweight Image Captioning based on CLIP and GPT2 encodings

Ansh Arora (2019022), Daksh Thapar (2018137),

Hardik Garg (2019040)

Problem definition

• Image Captioning is one of the most fundamental Vision-Language tasks, which involves the model generating a Natural Language description for a given image.
• SoTA approaches of the field, such as OSCAR and LEMON, while very powerful, are built on a huge number of parameters and thus are expensive to run.
• We propose a novel method that uses a fraction of these parameters and gives considerable performance in the same ballpark as the aforementioned models by utilizing OpenAI’s CLIP visual encodings and OpenAI’s GPT2 textual context encodings together.

Related works

Deep Captioning With Multimodal Recurrent

Neural Networks (M-RNN)

• It was one of the earliest methods, which made use of Multimodal Recurrent Neural Networks for captioning.
• Consists of two sub-networks - a deep recurrent neural network for sentence generation and a deep convolutional network for images.
• The RNN learns a dense feature embedding for each word in the dictionary and thus stores the semantic, temporal context generated in its recurrent layers. The CNN - generates the image representation. The multimodal part connects the language and vision models by a single layer representation.
• The activation of the three layers are inputted to the same multimodal feature space and added together to obtain the activation of the multimodal layer.

Related works

Show and Tell (2014) and Show Attend and Tell (2015)

• In the Show and Tell paper, a Deep CNN is utilized to generate representations of the image in the form of a fixed length embedding which are then inputted into the RNN decoder meant to generate sentences. The LSTM based RNN is fed in with words of the sentence, after which it predicts a caption.
• In the Show, Attend and Tell paper, the idea of Attention to make sure that the image’s most salient features are considered for captioning was introduced. Features extracted from lower convolutional layers are utilized to create annotation vectors for images, which are then fed into the decoder.

Related works

OSCAR: Object-Semantics Aligned Pre-training for Vision-Language Tasks

• Foundation: salient objects in an image are often mentioned in paired text
• Input: < word tokens, object tags, region features >

​

​

​

Related works

CLIP - Learning Transferable Visual Models From Natural Language Supervision

• The CLIP model learns relationships between the whole sentence and the image that it describes, on which it was trained with contrastive loss. Given the extensive training of the model on data with context, it thus acts like a great classifier as well, with a large amount of pre-learned features.

Related works

GPT2 - Language Models are Unsupervised Multitask Learners

• GPT2 is a large trained transformer based language model that has been extensively trained on nearly 40GB worth of data on 1.5Bn parameters.
• It was trained to predict the next word, given all of the previous words within some text.

​

Methodology

Our proposed Model -

​

Methodology

• First, to extract features from the image, we run pre-trained CLIP model on a given training image. This generates an embedding for the train image.
• The embedding is then parsed through a mapping network, where the CLIP generated embeddings are mapped to embedding vectors to be inputted into the language model.
• The aforementioned mapped visual embeddings are concatenated to the caption embeddings. These concatenated embeddings are then fed into the language model.
• The mapping network, which forms the only trainable sector of our architecture is trained by utilizing cross entropy loss.

Methodology

• The architecture we use for our mapping network is the Transformer architecture, which allows for better global attention between the input tokens.
• This transformer network is input with the CLIP generated visual encodings.

Methodology

• In our experiments, to see if our relatively smaller model performs well on the reduced dataset that we considered, we took the model from Show, Attend and Tell as the control for our experiment. This was done because of the fact that it formed a classical baseline and had around the same number of parameters as our model (a bit more in fact). �
• Both the models were ran on the same dataset for the same number of epochs, albeit with different hyperparameter choices that performed best on the two models. The control model’s empirical choices of hyperparameters were taken for the experiments.

Dataset detail and Results

• The dataset that we utilized was a subset of the MSCOCO 2014 dataset. We made use of the Karpathy splits for the captioning of the data made available by CSAIL.�
• The data was converted into 2 formats - one to be taken as input into our control model Show, Attend and Tell which utilizes tokens, the other to be taken as input to our model’s CLIP section which takes in raw sentences. �
• The final subset we considered for our experiments consisted of 7500 images for our train set and 2500 images for our val/test set.

Dataset EDA

Average Sentence Length: 10.45695170060774

Train Images: 7500

Train Captions: 37516

Validation Images: 2500

Validation Captions: 12508

Analysis-Results

The results we obtained were as follows, with due consideration given to the number of parameters of both models and the training times observed for the two.

​

​

​

��Oscar (135M parameters)

Show-Attend-Tell (57M parameters, 443 minutes)

CLIP model (53 M parameters, 290 minutes)

Loss Plots

Member Contributions

Equal Contribution from each member of the team

2 papers are implemented (Show-Attend-Tell-> Model 1, Clip Approach-> Model 2)

• Ansh Arora : Literature Review, Arch Model 2, Preprocessing of data and model 1
• Daksh Thapar : Literature Review, Arch Model 2, Preprocessing of data and model 2
• Hardik Garg : Literature Review, Arch Model 2, Preprocessing of data and model 2

We were all in the campus, so all the work was done together.

​

References

• Hu, X., Yin, X. and Lin, K. (2021). VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning. AAAI 2021 Computer Vision and Pattern Recognition.
• Xu, K., Ba, J. and Kiros, R. (2016). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. Machine Learning Computer Vision and Pattern Recognition.
• Luo, Y. and Ji, J. (2021). Dual-Level Collaborative Transformer for Image Captioning.
• Anderson, P. (2018). Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. CVPR Computer Vision and Pattern Recognition.
• Li, X. and Yin, X. (2020). Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. ECCV 2020.
• Radford, A. (2021). Learning Transferable Visual Models From Natural Language Supervision. CVPR 2021.