Nima Kalantari

CSCE 448/748 - Computational Photography

​

​

Stereo

Many slides from Steve Seitz

Depth of a scene

Credit: PetaPixel

Depth of a scene

Credit: Li & Snavely

Depth ambiguity

Depth ambiguity

• Structure and depth are inherently ambiguous from single views.

Images from Lana Lazebnik

Forced Perspective

https://www.youtube.com/watch?v=QWMFpxkGO_s

Stereo Vision

• Not that important for humans, especially at longer distances. Perhaps 10% of people are stereo blind.
• Many animals don’t have much stereo overlap in their fields of view

What cues help us to perceive 3d shape and depth?

​

Shading

[Figure from Prados & Faugeras 2006]

Shading

Credit: scientificamerican.com

Texture

[From A.M. Loh. The recovery of 3-D structure using visual texture patterns. PhD thesis]

Focus/defocus

[figs from H. Jin and P. Favaro, 2002]

Images from same point of view, different camera parameters

3d shape / depth estimates

Motion

Figures from L. Zhang

http://www.brainconnection.com/teasers/?main=illusion/motion-shape

Stereo photography

Invented by Charles Wheatstone 1838

Stereo photography

Credit: http://www.johnsonshawmuseum.org/

​

​

Basic Idea

Credit: https://people.well.com/user/jimg/stereo/stereo_list.html

Stereo

scene point

optical center

image plane

Stereo

Basic Principle: Triangulation

• Gives reconstruction as intersection of two rays

• Requires
• camera pose (calibration)
• point correspondence

Assume parallel optical axes, known camera parameters (i.e., calibrated cameras). What is expression for Z?

Similar triangles (p_l, P, p_r) and (O_l, P, O_r):

​

​

​

​

​

Geometry for a simple stereo system

disparity

+

̶

Depth from disparity

image I(x,y)

image I´(x´,y´)

Disparity map D(x,y)

(x´,y´)=(x+D(x,y), y)

So if we could find the corresponding points in two images, we could estimate relative depth…

General case, with calibrated cameras

• The two cameras need not have parallel optical axes.

Vs.

• Given p in left image, where can corresponding point p’ be?

Stereo correspondence constraints

Stereo correspondence constraints

Geometry of two views constrains where the corresponding pixel for some image point in the first view must occur in the second view.

• It must be on the line carved out by a plane connecting the world point and optical centers.

​

Epipolar constraint

​

• Epipolar Plane

Epipole

Epipolar Line

Baseline

Epipolar geometry

​

Epipole

http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html

​

​

​

Epipolar plane: plane containing baseline and world point

Epipole: point of intersection of baseline with image plane

Epipolar line: intersection of epipolar plane with the image plane

Baseline: line joining the camera centers

Epipolar constraint

​

This is useful because it reduces the correspondence problem to a 1D search along an epipolar line.

Image from Andrew Zisserman

Example

What do the epipolar lines look like?

O_l

O_r

O_l

O_r

1.

2.

Example: converging cameras

Figure from Hartley & Zisserman

Figure from Hartley & Zisserman

Example: parallel cameras

Where are the epipoles?

The correspondence problem

Epipolar geometry constrains our search, but we still have a difficult correspondence problem.

Basic stereo matching algorithm

• For each pixel x in the first image
• Find corresponding epipolar scanline in the right image
• Examine all pixels on the scanline and pick the best match x’
• Compute disparity x-x’ and set depth(x) = fB/(x-x’)

Correspondence search

• Slide a window along the right scanline and compare contents of that window with the reference window in the left image
• Matching cost: SSD or normalized correlation

Matching cost

disparity

Left

Right

scanline

Correspondence search

Left

Right

scanline

SSD

Correspondence search

Left

Right

scanline

Norm. corr

Correspondence problem

Source: Andrew Zisserman

Parallel camera example: epipolar lines are corresponding image scanlines

Correspondence problem

Neighborhoods of corresponding points are similar in intensity patterns.

Source: Andrew Zisserman

Correlation-based window matching

Source: Andrew Zisserman

Textureless regions

Textureless regions are non-distinct; high ambiguity for matches.

Source: Andrew Zisserman

Effect of window size

Source: Andrew Zisserman

Effect of window size

W = 3

W = 20

• Smaller window

+ More detail

• More noise

​

• Larger window

+ Smoother disparity maps

• Less detail

​

Results with window search

Window-based matching

Ground truth

Data