1 of 27

Clearer Frames, Anytime: Resolving Velocity Ambiguity in Video Frame Interpolation

Zhihang Zhong¹, Gurunandan Krishnan², Xiao Sun¹, Yu Qiao¹, Sizhuo Ma², Jian Wang²

¹Shanghai AI Lab, ²Snap Inc.

10/08/2023

2 of 27

Outline

Introduction: Video frame interpolation
Problem: Velocity ambiguity in time indexing
Methodology: Strategies for disambiguation
Experiment: Effectiveness of plug-and-play strategies
New feature: Manipulated interpolation of anything
Conclusion and future work
Appendix

2

3 of 27

Introduction: Video frame interpolation

Video frame interpolation (VFI) has wide applications

3

Slow motion of highlights

Sync video to the beat

Assisting video generation

Video compression

4 of 27

Introduction: Paradigms

Traditional flow-based approaches: linear motion; holes
Learning-based approaches include fixed-time & arbitrary-time interpolation
Arbitrary-time: faster for any timestep; no accumulation errors

4

arbitrary

time

5 of 27

Problem: Velocity ambiguity in time indexing

The velocities of individual objects within starting and ending frames remain undefined, introducing a velocity ambiguity, a myriad of plausible time-to-location mappings during training

As a result, models trained with time indexing tend to produce blurred and imprecise interpolations, as they average out the potential outcomes.

5

6 of 27

Problem: Velocity ambiguity in time indexing

Velocity ambiguity encompasses speed ambiguity & directional ambiguity

6

7 of 27

Methodology: Strategies for disambiguation

7

8 of 27

Methodology: Distance indexing

8

🡪

9 of 27

Methodology: Iterative reference-based estimation

Although distance indexing (a) addresses the scalar speed ambiguity, the directional ambiguity of motion remains a challenge.
We introduce an iterative reference-based estimation strategy (b),which incrementally estimates distances, beginning with nearby points and advancing to farther ones, to mitigate the remained ambiguity

9

10 of 27

Methodology: Plug-and-play

Our approach addresses challenges that are not bound to specific network architectures. Indeed, it can be applied as a plug-and-play strategy that requires only modifying the input channels for each model

10

11 of 27

Experiments: Vimeo90K septuplet dataset

Consists of 91,701 seven frame sequences with fixed resolution 448 x 256, extracted from 39,000 selected video clips

11

12 of 27

Experiment: State-of-the-art models and marks

[ECCV 2022] Real-Time Intermediate Flow Estimation for Video Frame Interpolation
[CVPR 2022] IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation
[CVPR 2023] Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation (EMA-VFI)
[CVPR 2023] AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation
[T] time indexing; [D] distance indexing; [R] reference-based estimation

12

13 of 27

Experiment: Qualitative

14 of 27

Experiment: Qualitative

15 of 27

Experiment: Convergence curves

[D] and [R] facilitate the convergence of each model

15

16 of 27

Experiment: Quantitative

16

In Table \ref{tab:vimeo90k_septuplet}, we provide a performance breakdown for each model variant. The models $[D]$ and $[D,R]$ in the upper half utilize ground-truth distance guidance, which is not available at inference in practice. The goal here is just to show the achievable upper-bound performance. On both pixel-centric metrics such as PSNR and SSIM, and perceptual measures like LPIPS and NIQE, the improved versions $[D]$ and $[D,R]$ outperform the base model $[T]$. Notably, the combined model $[D,R]$ using both distance indexing and iterative reference-based estimation strategies performs superior in perceptual metrics, particularly NIQE. The superior pixel-centric scores of model $[D]$ compared to model $[D,R]$ can be attributed to the indirect estimation (2 iterations) in the latter, causing slight misalignment with the ground-truth, albeit with enhanced details.

In realistic scenarios where the precise distance map is inaccessible at inference, one could resort to a uniform map akin to time indexing. The bottom segment of Table \ref{tab:vimeo90k_septuplet} shows the performance of the enhanced models $[D]$ and $[D,R]$, utilizing identical inputs as model $[T]$. Given the misalignment between predicted frames using a uniform distance map and the ground-truth, the enhanced models do not outperform the base model on pixel-centric metrics. However, we argue that in most applications, the goal of VFI is not to predict pixel-wise aligned frames, but to generate plausible frames with high perceptual quality. Furthermore, pixel-centric metrics are less sensitive to blur \citep{zhang2018unreasonable}, the major artifact introduced by velocity ambiguity. The pixel-centric metrics are thus less informative and denoted in gray. On perceptual metrics (especially NIQE), the enhanced models significantly outperforms the base model. This consistency with our qualitative observations further validates the effectiveness of distance indexing and iterative reference-based estimation.

17 of 27

Experiment: Quantitative

17

18 of 27

Experiment: User study

Questionnaire statistics of VFI model’s performance (Webapp)
Ranking of [T], [D], [T,R], and [D,R]
30 anonymous participants

19 of 27

Experiment: User study

The results align with our qualitative and quantitative findings. The [D,R] model variant emerged as the top-rated, underscoring the effectiveness of our strategies

20 of 27

New feature: Manipulated interpolation of anything

Instead of using a uniform map, it is also possible to use a spatially-varying 2D map as input to manipulate the motion of objects. Paired with SOTA segmentation models such as SAM, this empowers users to freely control the interpolation of any object, e.g., making certain objects backtrack in time

21 of 27

New feature: Manipulated interpolation of anything

Uniform interpolation

Manipulated mask

Inverse distance within mask

Set 0 for the rest

22 of 27

New feature: Manipulated interpolation of anything

22

Manipulation

Uniform

23 of 27

New feature: Demo of webapp

24 of 27

Conclusion and future work

We propose distance indexing and iterative reference-based�estimation to address the velocity ambiguity and enhance the capabilities of arbitrary time interpolation models

We present an unprecedented manipulation method that allows for customized interpolation of any object

Using multiple frames to estimate an accurate distance ratio map for a specific object is one of future works

24

25 of 27

Appendix: Additional experiments

Additional comparison of qualitative results. [T] RIFE (Tri) denotes RIFE trained in a fixed time indexing paradigm (Vimeo90K triplet dataset). [D] RIFE (Tri) denotes the corresponding model trained using distance indexing.

25

Using RIFE \citep{huang2022real} as a representative example, we extend our comparison to the fixed-time training paradigm, depicted in Figure \ref{fig:additional_qualitative_comparison}. The label $[T]$ RIFE (Tri) refers to the model trained on the Triplet dataset from Vimeo90K \citep{xue2019video} employing time indexing. Conversely, $[D]$ RIFE (Tri) indicates training on the same Triplet dataset but utilizing our distance indexing approach. Both $[D]$ RIFE and $[D,R]$ RIFE models are trained on the Septuplet dataset, consistent with our earlier comparison. Despite being trained on varied datasets, it is evident that the arbitrary time model outperforms the fixed time model. However, the efficacy of distance indexing appears restrained within the fixed-time training paradigm. This limitation stems from the fact that deriving distance representation solely from the middle frame yields a sparse distribution, making it challenging for the network to grasp the nuances of distance. We delve deeper into the quantitative analysis of these findings in Table \ref{tab:vimeo90k_triplet}. Compared to training arbitrary time models on the Septuplet dataset, the advantages of distance indexing become notably decreased when training fixed time models on the Triplet dataset.

26 of 27

Appendix: Additional experiments

26

27 of 27

Appendix: Limitations

The current system only allows for manipulations along a fixed trajectory

The overall effectiveness and accuracy of the model are intrinsically tied to the precision of optical flow estimation and the quality of segmentation masks

We use a uniform distance map which assumes every object moves at constant speed along their trajectories. In practice, different objects may have different acceleration

27

Despite the promising advancements introduced by our strategies, there are a few inherent limitations. Firstly, the current system only allows for manipulations along a fixed trajectory. It does not possess the capability to model the distribution of possible trajectories. Additionally, the overall effectiveness and accuracy of the model are intrinsically tied to the precision of optical flow estimation and the quality of segmentation masks (optional). Any inaccuracies or shortcomings in these preliminary steps can cascade and potentially impact the final interpolation results.

In our current approach, we use a uniform distance map which assumes every object moves at constant speed along their trajectories. In practice, different objects may have different acceleration and therefore this constant speed assumption may lead to suboptimal frame interpolation. However, estimating the acceleration merely from two frames is a challenging task. One potential future direction is to use multiple consecutive frames to capture the different acceleration of each object like \citet{xu2019quadratic}, which is then used to compute the precise distance ratio for each of them. This may further boost the performance and even beat time indexing methods on pixel-wise metrics during inference.