1 of 37

Shading-Aware Multi-view Stereo

Fabian Langguth, Kalyan Sunkavalli, Sunil Hadap, Michael Goesele

ECCV 2016

​

Presented by: Edward Zhang

UW CSE GRAIL Seminar

October 2016

2 of 37

Overview

  • Context: MVS
  • Precursor: Semerjian, ECCV 2014
  • This work: Langguth et al., ECCV 2016

3 of 37

The Multi-view Stereo Problem

4 of 37

Typical MVS Approach

Compute per-image depth maps

5 of 37

Epipolar Geometry

6 of 37

Epipolar Geometry

7 of 37

Epipolar Geometry

8 of 37

Epipolar Geometry

9 of 37

Epipolar Geometry

10 of 37

Epipolar Geometry

11 of 37

Epipolar Geometry

12 of 37

Epipolar Geometry

13 of 37

Photoconsistency

How to measure similarity of two patches? =

x y

?

14 of 37

Photoconsistency

How to measure similarity of two patches? =

x y

Low Sum of Squared Differences (SSD):

Low Sum of Absolute Differences (SAD):

Low Normalized SSD:

equivalent to high Normalized Cross Correlation (NCC)

?

15 of 37

Smoothness

Each depth value is independently computed

Real-world surfaces are “smooth”

16 of 37

A New Variational Framework for Multiview Surface Reconstruction

Semerjian, ECCV 2014

Contribution

  • Photoconsistency: NSAD = Gradient
  • Smoothness: Bicubic patches

​

17 of 37

Limit of NSAD

SAD

18 of 37

Limit of NSAD

“Normalized” SAD

19 of 37

Limit of NSAD

“Normalized” SAD

​

​

In 2D:

x0

s

s

20 of 37

Limit of NSAD

“Normalized” SAD

Equivalent to difference of gradients

21 of 37

Smoothness via Continuous Surface Representation

22 of 37

Shading-Aware Multi-View Stereo

Langguth et al., ECCV 2016

Contribution:

  • Use gradient to dial in Shape-from-Shading term

23 of 37

Shape from Shading

Uniform material

Consistent lighting

​

All color variation is due to shape:

Intensity is a function of normal

24 of 37

Retinex Theory

Small gradients from illumination, large gradients from texture

25 of 37

Retinex Theory

Small gradients from illumination, large gradients from texture

26 of 37

Retinex Theory

Small gradients from illumination, large gradients from texture

27 of 37

Retinex Theory

Small gradients from illumination, large gradients from texture

28 of 37

Why SfS?

  • Stereo is good at large gradients, bad at small gradients
    • Sharp edges are easier to match
    • Smooth regions will all have high similarity
  • SfS is good at small gradients, bad at large gradients
    • Sharp edges (albedo changes, shadows) violate SfS assumptions
    • SfS designed for smoothly varying surfaces

29 of 37

Energy Function

​

​

Stereo matching cost Shape-from-shading cost

(but only if gradient is small)

30 of 37

Shape-from-Shading Term

​

​

Difference between Observed and Predicted gradient

31 of 37

Lighting Model

​

​

​

Reflected Light is Albedo times Shading

Shading is a function of normal and incoming light

32 of 37

Albedo Invariance

SfS term kicks in with small gradient, i.e. constant albedo

Constant albedo disappears from

33 of 37

Albedo Invariance

SfS term kicks in with small gradient, i.e. constant albedo

Constant albedo disappears from

34 of 37

Results

35 of 37

36 of 37

37 of 37