A Review of Video Generation Approaches

Rishika Bhagwatkar, Saketh Bachu, Khurshed Fitter, Akshay Kulkarni, Shital Chiddarwar

PICC 2020-Paper ID: 401

ICETEST 2020

December 17 - 19, 2020

GEC Thrissur

INTRODUCTION

• The task of video generation is complicated but has gained a high traction
• Useful for tasks such as scene generation
• Existing methods use pixel transformations, stochastic modelling, variational deep learning, etc
• We intend to present an overview of these existing methods

1

PICC 2020

December 17 - 19, 2020

GEC Thrissur

MODELLING PIXEL TRANSFORMATIONS

• Goodfellow et al. [1] introduced an unsupervised method involving pixel transforms.

​

• Backbone is LSTM, further modifications introduce DNA, CDNA and STP.

​

• Achieved satisfactory results for at least 10 time steps

2

PICC 2020

December 17 - 19, 2020

GEC Thrissur

STOCHASTIC VIDEO GENERATION

• Involves the usage of time dependant stochastic latent variables. Two variants are SVG-FP and SVG-LP.

​

• The model architecture is based on an LSTM interposed between an encoder and a decoder.

​

• Lee et al.[3] proposed SAVP using Conv-LSTM.

​

​

3

PICC 2020

December 17 - 19, 2020

GEC Thrissur

STOCHASTIC VIDEO GENERATION

• The major difference between SVG-LP [2] and SAVP [3] is the loss function employed in both the cases.

​

• In Kumar et al. [4], they use a flow-based generative model with a multi scale architecture as a building block.

4

PICC 2020

December 17 - 19, 2020

GEC Thrissur

TRANSFORMER BASED APPROACHES

• Various Natural Language Processing tasks have achieved high fidelity using Attention and Self-Attention mechanisms.

​

• In Axial Transformer, attention is applied along a single axis of multi-dimensional tensors.

​

5

PICC 2020

December 17 - 19, 2020

GEC Thrissur

TRANSFORMER BASED APPROACHES

• Video Transformer introduce a 3D, block-local self-attention mechanism.

​

• Latent Video Transformer model upon discrete latent space to alleviate GP utilisation, memory footprint and to escalate sampling frequency.

6

PICC 2020

December 17 - 19, 2020

GEC Thrissur

GANs BASED APPROACHES

• Previous works of GANs considered the the texture and spatial consistency of objects separately from their temporal dynamics.

​

• MoCoGAN maps a sequence of random vectors, to a sequence of frames based on a prior.

​

7

PICC 2020

December 17 - 19, 2020

GEC Thrissur

GANs BASED APPROACHES

​

• Similar to MoCoGAN, DVD-GAN utilise 2 discriminators.

​

• Spatial Discriminator and Temporal Discriminator

​

• Uses Recurrent Neural Networks in the generator.

8

PICC 2020

December 17 - 19, 2020

GEC Thrissur

GANs BASED APPROACHES

​

• TrIVD-GAN proposes a new architecture of discriminator decomposition .

​

• Achieves better convergence and performance and proposed new recurrent units based on transformation.

9

PICC 2020

December 17 - 19, 2020

GEC Thrissur

REFERENCES

1. Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction, 2016
2. Emily Denton and Rob Fergus. Stochastic video generation with a learned prior, 2018
3. Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction, 2018
4. Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A conditional flow-based model for stochastic video generation, 2020

​

10

PICC 2020

December 17 - 19, 2020

GEC Thrissur

Thank you!

11

PICC 2020

December 17 - 19, 2020

GEC Thrissur

PICC 2020-Paper ID: 401