1 of 25

SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow

Yihan Wang, Lahav Lipson, Jia Deng

2 of 25

Optical Flow: A Fundamental Vision Task

Frame t

Frame t+1

Optical Flow: a 2D vector field

3 of 25

Applications of Optical Flow

Visual Odometry

Video Generation

Neural Rendering

4 of 25

SEA-RAFT: A New Model for Optical Flow

  • Simple
    • Simplified RAFT architecture
  • Efficient:
    • 2.3x ~ 24x faster
  • Accurate:
    • 23% error reduction on Spring

​

5 of 25

SEA-RAFT: Contributions

  • Loss Design

​

  • Model Architecture

​

  • Rigid-Flow Training

6 of 25

SEA-RAFT: Contributions

  • Loss Design

​

  • Model Architecture

​

  • Rigid-Flow Training

7 of 25

 

Frame t

Frame t+1

 

Training error dominated by ambiguous cases

FlyingChairs dataset, from FlowNet: Learning Optical Flow with Convolutional Networks, Dosovitskiy et al., ICCV, 2015

8 of 25

 

 

 

 

 

9 of 25

 

 

 

 

 

 

 

 

 

 

Prob. loss allows expressing uncertainty

10 of 25

Choice of Prob. Loss Matters

 

 

 

11 of 25

Our Solution: Mixture-of-Laplace

 

 

 

 

 

 

12 of 25

Our Solution: Mixture-of-Laplace

 

 

 

 

 

 

 

13 of 25

Our Solution: Mixture-of-Laplace

 

 

 

 

 

 

Bounded, log-space regression for numerical stability

14 of 25

Mixture-of-Laplace: Efficient, Accurate

 

 

 

 

 

 

On Spring (sub-val)

 

15 of 25

SEA-RAFT: Contributions

  • Supervision

​

  • Model Architecture

​

  • Rigid-Flow Training

16 of 25

Review: RAFT for Optical Flow

RAFT: Recurrent All-Pair Field Transforms for Optical Flow ,Teed et al., ECCV, 2020

Feature Extraction

Iterative Refinements

17 of 25

Iterative Refinements are Expensive

82%

86%

Iterative refinements cost >80%

18 of 25

Our Solution: Direct Regression of the Initial Prediction

RAFT

Zero Initialization

19 of 25

Our Solution: Direct Regression of the Initial Prediction

RAFT

Zero Initialization

SEA-RAFT

Initial Prediction

20 of 25

Direct Regression: Simple, Efficient

RAFT

SEA-RAFT

86% cost

26% cost

 

21 of 25

SEA-RAFT: Contributions

  • Supervision

​

  • Model Architecture

​

  • Rigid-Flow Training

22 of 25

Definition: Rigid-Flow

The entire scene is rigid, no dynamic objects, only camera motion

Static scene

Camera Pose 2

Camera Pose 1

Frame 1

Frame 2

23 of 25

Rigid-Flow Pre-Training

306k rigid-flow from TartanAir, pre-train

123k optical flow, fine-tune

Tartanair: A dataset to push the limits of visual slam, Wang et al., IROS, 2020

24 of 25

Rigid-Flow Pre-Training: Better Generalization

  • GMFlow (not RAFT-style)
    • Zero-shot KITTI: -22% Fl-epe

​

  • SEA-RAFT:
    • Zero-shot Sintel: -13% Final EPE
    • Zero-shot KITTI: -4% Fl-epe
    • Zero-shot Spring: -4% 1px outlier

GMFlow: Learning Optical Flow via Global Matching, Xu et al., CVPR, 2021

25 of 25

SEA-RAFT: New Optical Flow Model

  •  

https://github.com/princeton-vl/SEA-RAFT

To appear in poster session 1

(1hr later!)