CNN-RNN BASED AUTOMATIC CAPTION GENERATION USING SPATIAL AND TEMPORAL FEATURES OF VIDEOS

By

ABHIRUP BHATTACHARYA

Examination Roll Number: M4SWE21018

Registration Number: 150050 OF 2019-2020

​

Under the Guidance & Supervision of

Mr. Somenath Dhibar

​

Assistant Professor

Department of Information Technology

Jadavpur University

2021

Introduction

• Ways of representation of Information has evolved with time, from text to image and from audio to audio-visual means.

​

​

• Real-world videos available to us in the Open Domain are diverse and have complex dynamics.

​

​

• For an AI, It is difficult to classify videos based on their content due to the diverse nature of objects, scenes, actions and their attributes.

​

​

• But with the advent of Deep Learning and some concepts of Natural Language Processing, classifying videos and generating descriptive captions has become possible.

Video Captioning

Def: “Video Captioning is a task of automatic captioning a video by understanding the action and event in the video which can help in the retrieval of the video efficiently through text.” [1]

​

Fig 1.0: High Level Diagrammatic Representation of Video Captioning

Applications of Video Captioning

• Self Driving Cars

​

• Video Recommendation System

​

• Human Robot Interaction

​

• Video Description for the Deaf

​

• Automatic Alarm Generation for Security System

​

The List can be endless since this is vast a field with numerous applications

​

​

​

​

Figure 2.0 : Self Driving car [2]

Figure 3.0 : Human Robot Interaction [3]

Previous attempts to Video Captioning

Below are some of the work done previously

​

• Video2text: Learning to annotate video content [4].

​

​

• Generating natural-language video descriptions using text-mined knowledge [5].

​

​

• Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shoot recognition [6].

​

​

• Integrating language and vision to generate natural language descriptions of videos in the wild [7].

​

​

• Translating videos to natural language using deep recurrent neural networks [8].

​

​

• Sequence to Sequence - Video to Text [9].

Methodology and Proposed Model

• Our proposed Model Consists of a CNN-RNN based Encoder–Decoder [10,11] architecture.

​

• We have taken Inspiration from the works done in Sequence to Sequence - Video to Text [9] to propose our models.

​

​

​

​

Extracted Features

Fig 4.0: Diagrammatic Representation of Video Captioning Process

Methodology and Proposed Model

​

​

​

​

• In all our models, we first extract frames from all the videos.

​

​

• Then we use a Pre-trained [12,13] CNN to extract features for the videos using the extracted frames.

​

​

• Then the video Feature along with the corresponding available pre-processed captions for training are fed as input to the Encoder-Decoder Model.

​

​

• The Encoder-Decoder architecture is implemented using a stack of two LSTMs [14]

​

​

• One LSTM acts as the Encoder while the other acts as the Decoder.

​

• The Encoder-Decoder model is responsible for learning and predicting text description for each video.

​

​

​

​

​

​

​

​

Methodology and Proposed Model

During Training

​

• The Extracted Frame Features are fed to the Encoder LSTM

​

​

• The output of the Encoder LSTM cells are rejected.

​

​

• Only the final state output of Encoder LSTM is considered

​

​

• The Decoder accepts the final state output of the Encoder as its initial input along with Tokenized [15] Captions

​

​

• The number of LSTM cells in the Encoder is the number of frames extracted and the number of LSTM cells in the Decoder is maximum size of the sentence that can be generated.

​

Methodology and Proposed Model

Fig 5.0: Training Model [16]

Methodology and Proposed Model

​

​

​

​

​

​

​

​

During Testing

​

• Using the extracted frames, video features are extracted and passed on to the Encoder LSTM

​

​

• The Final State of Encoder LSTM is passed on to the Decoder as input along with <bos> tag indicating beginning of the sentence.

​

​

​

• The Decoder now starts Predicting the captions till <eos> is encountered.

​

Methodology and Proposed Model

Fig 6.0 : Inference Model [16]

Methodology and Proposed Model

We have proposed four different models based on the type of features they were trained with:

​

1. Model Trained using Spatial Features

​

​

• Model Trained using Optical Flow features

​

​

• Model Trained using Mean of Spatial and Optical Flow Features

​

​

• Model Trained using Concatenated Spatial and Optical Flow Features

​

​

​

​

​

​

Model Trained using Spatial Features

• First we extract 80 frames from each of the videos.

​

• Then for each video, we send all the frames to a pre-trained CNN VGG-16 model [17].

​

• We extract the spatial features from the second fully connected layer (fc2) of the VGG-16 model.

​

• The result is a feature vector of dimension 80 x 4096. Where 80 is the number of frames and 4096 is the number of features for each frame.

​

• We have used LSTM as the preferred RNN to implement our Encoder-Decoder Architecture.

​

• Now, the features of the first frame are fed into the 1st LSTM cell of the encoder, followed by the features of the second frame to the 2nd LSTM cell and similarly all the 80 frames are fed as input to respective LSTM cells

Model Trained using Spatial Features

• The number of LSTM cells set for the encoder is same as the number of frames considered for our work.

​

​

• The output of the LSTM cells in the encoder are discarded.

​

​

• Only the final state of the encoder is considered and is fed as input to initial state of the decoder.

​

​

• In the first decoder LSTM , the < bos > acts as input to start the sentence. Each and every word from the captions in the training set is fed as input to the decoder till < eos >.

​

​

• The encoder time steps is the number of LSTM cells we used for the encoder and was set to 80 (since 80 frames).

​

Model Trained using Spatial Features

• Encoder tokens is the number of features that each frame has and was set to 4096.

​

• The time steps for the decoder is the number of LSTM cells we use for the decoder and was set to 10 (since we have limited our caption size to maximum 10 words) and number of decoder tokens is the size of the vocabulary and was set to 1500.

​

• After training the entire model, the encoder and the decoder models are saved separately.

​

​

• For testing, all the 80 features are passed on to the encoder one by one.

​

​

• The final state of the encoder is taken and is input to the decoder along with the < bos > token after which the decoder starts to predict the captions.

​

Model Trained using Optical Flow features

• First we extract 81 frames from each of the videos.

​

• Then for each video, we use all the 81 frames to compute Dense Optical Flow and obtain flow images.

​

• Then for each video, we send all the flow images to a pre-trained CNN Inception V3 model [18].

​

• We extract the Optical flow features from the pool3 layer of the Inception V3 model.

​

• The result is a feature vector of dimension 81 x 2048. Where 81 is the number of frames and 2048 is the number of features for each frame.

​

• We have used LSTM as the preferred RNN to implement our Encoder-Decoder Architecture.

​

• Now, the features of the first frame are fed into the 1st LSTM cell of the encoder, followed by the features of the second frame to the 2nd LSTM cell and similarly all the 81 frames are fed as input to respective LSTM cells

​

​

Model Trained using Optical Flow features

• The number of LSTM cells set for the encoder is same as the number of frames considered for our work.

​

​

• The output of the LSTM cells in the encoder are discarded.

​

​

• Only the final state of the encoder is considered and is fed as input to initial state of the decoder.

​

​

• In the first decoder LSTM , the < bos > acts as input to start the sentence. Each and every word from the captions in the training set is fed as input to the decoder till < eos >.

​

​

• The encoder time steps is the number of LSTM cells we used for the encoder and was set to 81 (since 81 frames)

​

Model Trained using Optical Flow features

• Encoder tokens is the number of features that each frame has and was set to 2048.

​

• The time steps for the decoder is the number of LSTM cells we use for the decoder and was set to 10 (since we have limited our caption size to maximum 10 words) and number of decoder tokens is the size of the vocabulary and was set to 1500

​

• After training the entire model, the encoder and the decoder models are saved separately.

​

​

• For testing, all the 81 features are passed on to the encoder one by one.

​

​

• The final state of the encoder is taken and is input to the decoder along with the

< bos > token after which the decoder starts to predict the captions.

Model Trained using Mean of Spatial and Optical Flow Features

• First we extract 81 frames from each of the videos.

​

• Then for each video, we use all the 81 frames to compute Dense Optical Flow and obtain flow images.

​

• Then for each video, we send all the flow images to a pre-trained CNN Inception V3 model.

​

• Again for each video, we send all the extracted frames to the pre-trained CNN Inception V3 model.

​

• We extract the Optical flow features and Spatial features from the pool3 layer of the Inception V3 model separately.

​

• We get two different vectors of dimension 81x2048 each. We do an element by element addition of the two vectors .

​

• On the resultant vector, we divide each element by 2 .

​

• The result is a feature vector of dimension 81 x 2048.

​

Model Trained using Mean of Spatial and Optical Flow Features

• We have used LSTM as the preferred RNN to implement our Encoder-Decoder Architecture

​

• Training and validation was done in the same manner as done for the model trained with Spatial features and the model trained with Optical flow features.

Model Trained using Concatenated Spatial and Optical Flow Features

• First we extract 81 frames from each of the videos.

​

• Then for each video, we use all the 81 frames to compute Dense Optical Flow and obtain flow images.

​

• Then for each video, we send all the flow images to a pre-trained CNN Inception V3 model.

​

• Again for each video, we send all the extracted frames to the pre-trained CNN Inception V3 model.

​

• We extract the Optical flow features and Spatial features from the pool3 layer of the Inception V3 model separately.

​

• Then both the vectors generated were concatenated with each other and the result is another vector which is of the form of 81x4096

​

• We have used LSTM as the preferred RNN to implement our Encoder-Decoder Architecture

​

Model Trained using Concatenated Spatial and Optical Flow Features

• The number of time steps for the Encoder (number of LSTMs in the Encoder) was 81 and the number of Encoder Tokens were set to 4096.

​

• However, the decoder was same like in case of the other models.

​

• Training and validation was done in the same manner as done for the model trained with Spatial features , the model trained with Optical flow features and the model trained with mean of Spatial and Optical flow features.

​

Loss During Training and Validation

Fig 10.0: Model Trained with Concatenated Spatial and Optical Flow Features

Fig 9.0: Model Trained with Mean of Spatial and Optical Flow Features

Fig 8.0: Model Trained with Optical Flow Features

Fig 7.0: Model Trained with Spatial Features

Model Accuracy During Training and Validation

Fig 11.0: Model Trained with Spatial Features

Fig 12.0: Model Trained with Optical Flow Features

Fig 13.0: Model Trained with Mean of Spatial and Optical Flow Features

Fig 14.0: Model Trained with Concatenated Spatial and Optical Flow Features

For the sake of simplicity lets denote the captions generated by the below models as:

​

Model Trained with Spatial Features: Model A

​

Model Trained with Optical Flow Features: Model B

​

Model Trained with Mean of Spatial and Optical Flow Features: Model C

​

Model Trained with Concatenated Spatial and Optical Flow Features: Model D

​

​

​

​

​

​

​

​

Results

Results

Some of the results that were good:

​

​

​

​

​

​

​

​

​

Actual Captions: A cat is playing a piano

Model A: a man is playing a .

Model B: a woman is peeling a apple Model C: a cat is playing a.

Model D: a cat is playing the piano.

Actual Captions: A guy is riding a motorcycle.

Model A: a man is riding a motorcycle

Model B: a woman is frying something.

Model C: a man is riding a motorcycle

Model D: a woman is riding a motorcycle

Fig 15.1: Good Results

Fig 15.2: Good Results

Actual Captions: A boy is playing guitar.

Model A: a man is playing the guitar

Model B: a man is playing.

Model C: a boy is playing a guitar

Model D: a man is playing a guitar

Actual Captions: A bear is walking on the rock.

Model A: a man is being

Model B: a baby is crying.

Model C: a polar bear is walking

Model D: a polar bear is walking

Results

Fig 16.1: Good Results

Fig 16.2: Good Results

Results

Some of the results that were not good:

​

​

​

​

​

​

​

​

​

Actual Captions: A girl is jumping rope

Model A: a man is doing a

Model B: a man is riding a.

Model C: a man is walking

Model D: a man is running

Actual Captions: A worker on an electrical pole is shocked.

Model A: a man is running

Model B: a man is making a box.

Model C: a man is running on a rope

Model D: a man is riding a

Fig 17.1: Poor Results

Fig 17.2: Poor Results

Performance Analysis

We have evaluated the BLEU@4 score and the Meteor Score for all the four models.

Fig 18.0: BLEU@4 score representation of all four models

Performance Analysis

Fig 19.0: METEOR score representation of all four models

Performance Analysis

Fig 20.0: Performance Analysis of our models with others based on BLEU@4 scores

Fig 21.0: Performance Analysis of our models with others based on METEOR scores

[7]

[7]

[8]

[8]

[8]

[8]

[8]

[8]

[8]

[8]

Performance Analysis

• We have compared the performance of our models with the models in [7] and [8].

​

• The Comparison was possible because of the fact that all three approaches use the same dataset Microsoft Video Description Corpus (MSVD) [19].

​

• It was observed that our models performs better than the models in [7].

​

• However, our models underperforms when compared with the models in [8].

​

• The Models in [8] used a CNN-RNN Based Encoder Decoder architecture and have used LSTM as their preferred RNN as the decoder.

​

• They have used a CNN Caffe [20] as their preferred CNN to extract the frame features.

​

• The architecture of the model in [8] is similar to image captioning models.

​

Performance Analysis

The reason why their models [8] performed better than ours was because of the following:

​

​

• They have sampled the frames by taking every 10^th frame to extract the features. Whereas we took the first 80 frames only.

​

• Their model was previously trained on a huge dataset for image captioning and then was fine-tuned with the MSVD dataset. Whereas our models were only trained from scratch using MSVD dataset.

​

• The CNN chosen to extract features might also be another reason.

​

Conclusion

• We have proposed and implemented four different models based on the type of feature used to train them

​

​

• It has been observed that the models have generated close to perfect captions in a considerable number of cases.

​

​

• However it has also been observed that among all the models, the model trained with concatenated Spatial and Optical Flow features has performed comparatively better than the rest.

Future Scope

• In the future, we have plans of implementing the same using 3DCNN models and extend the entire concept towards making a product capable of detecting theft, robbery or shoplifting in departmental stores and shopping malls.

​

​

• The model would recognize the action and predict real time alerts from real time video capture mediums

References

• [1] https://paperswithcode.com/task/video-captioning

​

• [2] https://www.greenbiz.com/article/7-companies-steering-self-driving-car-craze

​

• [3] https://asegrad.tufts.edu/academics/explore-graduate-programs/human-robot-interaction

​

• [4] H. Aradhye, G. Toderici, and J. Yagnik. Video2text: Learning to annotate video content. In ICDMW, 2009.

​

• [5] N. Krishnamoorthy, G. Malkarnenkar, R. J. Mooney, K. Saenko, and S. Guadarrama. Generating natural-language video descriptions using text-mined knowledge. In AAAI, July 2013.

​

• [6] S. Guadarrama, N. Krishnamoorthy, G. Malkarnenkar, S. Venugopalan, R. Mooney, T. Darrell, and K. Saenko. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shoot recognition. In ICCV,Dec 2013.

​

​

​

References

​

​

​

• [7] J. Thomason, S. Venugopalan, S. Guadarrama, K. Saenko, and R. J. Mooney. Integrating language and vision to generate natural language descriptions of videos in the wild. In Coling, volume 2, page 9, 2014

​

• [8] S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. Mooney, and K. Saenko. Translating videos to natural language using deep recurrent neural networks. In NAACL, 2015.

​

• [9] S. Venugopalan, R. Marcus, D. Jeffrey, M. Raymond, D. Trevor, and S. Kate. Sequence to Sequence - Video to Text. In ICCV, 2015

​

• [10] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In NIPS, 2014

​

• [11] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio. ¨ On the properties of neural machine translation: Encoderdecoder approaches. arXiv:1409.1259, 2014

​

• [12] https://towardsdatascience.com/4-pre-trained-cnn-models-to-use-for-computervisionwith-transfer-learning-885cb1b2dfc

​

​

• [13] https://www.analyticsvidhya.com/blog/2017/06/transfer-learning-the-art-offinetuning-a-pre-trained

​

• [14] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8), 1997.

​

• [15] https://towardsdatascience.com/your-guide-to-natural-language-processingnlp48ea2511f6e1

​

• [16] https://medium.com/analytics-vidhya/video-captioning-with-keras-511984a2cfff

​

• [17] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 201

​

• [18] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens and Z. Wojna, ”Rethinking the Inception Architecture for Computer Vision,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2818-2826, doi: 10.1109/CVPR.2016.308.

References

References

​

• [19] D. L. Chen and W. B. Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL, 2011.

​

​

• [19] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. ACMMM, 2014

THANK

YOU