1 of 19

Dynamic Depth: Disentangling Object Motion and Occlusion for Unsupervised Multi-frame Monocular Depth

2 of 19

Monocular Depth Prediction

3 of 19

Unsupervised �Monocular Depth Prediction

Re-projection Loss is the key for unsupervised monocular depth prediction.

 

Pose Net

6-DOF Pose

Depth Net

 

 

Re-Projection Loss

4 of 19

Multi-frame �Monocular Depth Prediction

Cost volume is proved to be an effective way to leverage temporal frames to improve the overall depth quality, which is also based on the re-projection geometry.

 

Pose Net

Depth Encoder

 

 

Re-

Projection

Depth Decoder

Cost Volume

5 of 19

Re-projection Geometry

  •  

 

Re-Projection

 

 

Suppose to match

6 of 19

Dynamic Obj Mismatch problem

Dynamic objects will cause the ‘Mismatch’ problem.

 

Re-Projection

 

 

Obj Motion

 

 

Mismatch!

7 of 19

Dynamic Obj Occlusion problem

Dynamic objects will cause ‘Occlusion’ problem.

 

Re-Projection

 

 

Obj Motion

Mismatch!

Occlusion!

Occluded!

Visible!

 

8 of 19

  • ‘Mismatch’ and ‘Occlusion’ affects:
    • Re-projection loss (Self-supervision).
      • Existing solutions rely on the object motion prediction.
    • Cost volume(Temporal frames inference).
      • No existing solution.
  • We propose to alleviate these problems in BOTH loss function and Cost volume side, to enable the temporal reasoning in dynamic objects areas.

Motivation:

9 of 19

We Propose DynamicDepth:

Depth Prior Net

Pose Net

Depth

Encoder

 

 

 

 

Occlusion-aware

cost volume

Depth

Decoder

 

Dynamic Object Motion Disentanglement

(DOMD)

Dynamic Object Cycle Consistency Loss

Our Contribution:

    • Novel Dynamic Object Motion Disentanglement (DOMD) module.
    • Dynamic Object Cycle Consistent training scheme.
    • Occlusion-aware Cost Volume and Re-projection Loss

10 of 19

DOMD Module

Re-project the dynamic object patch with ‘depth prior’ prediction

 

 

Depth Prior Prediction

11 of 19

DOMD Module

Replace dynamic object patch with re-projected image patch.

 

DOMD

12 of 19

DOMD Module

This replacement will alleviate the ‘Mismatch’ problem.

 

Re-Projection

 

 

Obj Motion

 

Match

Occlusion

13 of 19

Occlusion-aware Cost Volume

…

 

Occlusion-aware

Cost Volume

 

 

Occlusion Filling

…

Sharing Weights

-

-

-

-

…

 

 

 

 

 

 

 

 

 

 

Warp by All Depth Hypothesis

 

 

 

 

The occluded areas are filled with non-occluded cost values.

14 of 19

Occlusion-aware Re-projection Loss

 

 

Re-proj Error at t-1

 

The occluded areas are filled with non-occluded cost values.

15 of 19

Occlusion-aware Re-projection Loss

Source Frame

: From visible frame

: From occluded frame

Widely Used Per-pixel min Loss

Re-proj Error at t-1

 

16 of 19

Occlusion-aware Re-projection Loss

Source Frame

Our Occlusion-aware Loss

: From visible frame

: From occluded frame

Re-proj Error at t-1

 

17 of 19

Conclusion: Our method outperformed all the other methods on the Cityscapes and KITTI dataset.

18 of 19

Conclusion: Our method significantly outperformed all the other methods especially on the Dynamic objects areas.

19 of 19