1 of 31

Nima Kalantari

CSCE 448/748 - Computational Photography

Light Fields

Many slides from Alexei Efros, James Hays, Mark Levoy, and Sen et al.

2 of 31

What is light?

Electromagnetic radiation (EMR) moving along rays in space

R(λ) is EMR, measured in units of power (watts)

λ is wavelength

Useful things:

Light travels in straight lines
In vacuum, radiance emitted = radiance arriving

i.e. there is no transmission loss

3 of 31

Light field

Scene

Light Field

4 of 31

The Plenoptic Function

Q: What is the set of all things that we can ever see?

A: The Plenoptic Function (Adelson & Bergen)

Let’s start with a stationary person and try to parameterize everything that he can see…

Figure by Leonard McMillan

5 of 31

Grayscale snapshot

is intensity of light

Seen from a single view point
At a single time
Averaged over the wavelengths of the visible spectrum

P(θ,φ)

6 of 31

Color snapshot

is intensity of light

Seen from a single view point
At a single time
As a function of wavelength

P(θ,φ,λ)

7 of 31

A movie

is intensity of light

Seen from a single view point
Over time
As a function of wavelength

P(θ,φ,λ,t)

8 of 31

Holographic movie

is intensity of light

Seen from ANY viewpoint
Over time
As a function of wavelength

P(θ,φ,λ,t,V_X,V_Y,V_Z)

9 of 31

The Plenoptic Function

Can reconstruct every possible view, at every moment, from every position, at every wavelength
Contains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality! Not bad for a function…

P(θ,φ,λ,t,V_X,V_Y,V_Z)

10 of 31

Image

Image plane

2D

position

11 of 31

2D: Image

All rays through a point

Panorama?

Slide by Rick Szeliski and Michael Cohen

12 of 31

Spherical Panorama

All light rays through a point form a ponorama

Totally captured in a 2D array -- P(θ,φ)

13 of 31

Sampling Plenoptic Function (top view)

Just lookup – Google Street View

14 of 31

Ray

Let’s not worry about time and color:

5D

3D position
2D direction

P(θ,φ,V_X,V_Y,V_Z)

Slide by Rick Szeliski and Michael Cohen

15 of 31

How can we use this?

Surface

Camera

No Change in

Radiance

Lighting

16 of 31

Ray Reuse

Infinite line

Assume light is constant (vacuum)

4D

2D direction
2D position
non-dispersive medium

Slide by Rick Szeliski and Michael Cohen

17 of 31

Light field

Scene

t

s

v

u

(u₁ , v₁)

(s , t)

L(u, v, s, t) =

L(u₁, v₁, s , t)

t

s

(u₂ , v₂)

L(u₂, v₂, s , t)

4D Light Field

Camera locations

Image pixels

(u , v)

18 of 31

Capturing process

19 of 31

Lumigraph / Lightfield

20 of 31

Stanford multi-camera array

640 × 480 pixels ×�30 fps × 128 cameras

synchronized timing
continuous streaming
flexible arrangement

21 of 31

Light field photography using a handheld plenoptic camera�Commercialized as Lytro

Ren Ng, Marc Levoy, Mathieu Brédif,

Gene Duval, Mark Horowitz and Pat Hanrahan

22 of 31

Conventional versus light field camera

23 of 31

Conventional versus light field camera

uv-plane

st-plane

24 of 31

Prototype camera

4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens

Contax medium format camera

Kodak 16-megapixel sensor

Adaptive Optics microlens array

125μ square-sided microlenses

25 of 31

Lumigraph / Lightfield

26 of 31

27 of 31

Digitally moving the observer

moving the observer = moving the window we extract from the microlenses

28 of 31

Digitally stopping-down (aperture change)

stopping down = summing only the central portion of each microlens

Σ

29 of 31

Digital refocusing

refocusing = summing windows extracted from several microlenses

Σ

30 of 31

Example of digital refocusing

31 of 31

Example of moving the observer