Favorite restaurant?

ⓘ Start presenting to display the poll results on this slide.

Quiz 5 (on Canvas)�Closed book / closed note

Ends at 1:08pm

Light & Perception

CS5670: Computer Vision

Reading

• Szeliski 2^nd Edition, Chapter 2.2

Announcements

• Project 3 code due Monday, March 20, by 8pm to GitHub Classroom
• Project 3 artifact (panorama) due Tuesday, March 21, by 8pm to CMSX
• Project 4 to be released on Tuesday, March 21, due Friday, March 31 by 8pm
• Final exam is planned to be in class during the last lecture on Tuesday, May 9

Project 4 demo

Can we determine shape from lighting?

• Are these spheres?
• Or just flat discs painted with varying color (albedo)?
• There is ambiguity between shading and reflectance
• But still, as humans we can understand the shapes of these objects

What we know: Stereo

Key Idea: use camera motion to compute shape

Next: Photometric Stereo

Key Idea: use pixel brightness to understand shape

Next: Photometric Stereo

Key Idea: use pixel brightness to understand shape

Photometric Stereo

Input

(1 of 12)

Normals (RGB colormap)

Normals (vectors)

Shaded 3D�rendering

Textured 3D�rendering

What results can you get?

Light

• Readings
• Szeliski, 2.2, 2.3

by Ted Adelson

Light

• Readings
• Szeliski, 2.2, 2.3

by Ted Adelson

Properties of light

• Today
• What is light?
• How do we measure it?
• How does light propagate?
• How does light interact with matter?

Radiometry

• What determines the brightness of a pixel?

Light source properties

Surface properties

Surface properties

Radiometry

https://www.instagram.com/p/BtgX55ZBhU-/

@robertwestonbreshears

• What determines the brightness of a pixel?

Radiometry

• What determines the brightness of a pixel?

Radiometry

Light source�properties

Surface �shape

Surface reflectance�properties

Optics

Sensor characteristics

Slide by L. Fei-Fei

Exposure

• What determines the brightness of a pixel?

What is light?

Electromagnetic radiation (EMR) moving along rays in space

• R(λ) is EMR, measured in units of power (watts)
• λ is wavelength

Light field

• We can describe all of the light in the scene by specifying the radiation (or “radiance” along all light rays) arriving at every point in space and from every direction

​

The plenoptic function describes all of this light:

The light field

​

• Known as the plenoptic function
• If you know R, you can predict how the scene would appear from any viewpoint.

​

The light field

• Assume radiance does not change along a ray
• what does this assume about the world?
• Parameterize rays by intersection with two planes:

​

​

​

​

​

• Usually drop λ and time parameters
• How could you capture a light field?

Capturing light fields

Stanford/Cornell spherical gantry

Stanford Multi-Camera Array

Lego Mindstorms Gantry

Handheld light field camera

Light field example

More info on light fields

• If you’re interested to read more:

​

• The plenoptic function
• Original reference: E. Adelson and J. Bergen, "The Plenoptic Function and the Elements of Early Vision," in M. Landy and J. A. Movshon, (eds) Computational Models of Visual Processing, MIT Press 1991.
• L. McMillan and G. Bishop, “Plenoptic Modeling: An Image-Based Rendering System”, Proc. SIGGRAPH, 1995, pp. 39-46.

​

• The light field
• M. Levoy and P. Hanrahan, “Light Field Rendering”, Proc SIGGRAPH 96, pp. 31-42.
• S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, "The lumigraph," in Proc. SIGGRAPH, 1996, pp. 43-54.

Color perception

Electromagnetic radiation (EMR) moving along rays in space

• R(λ) is EMR, measured in units of power (watts)
• λ is wavelength

Perceiving light

• How do we convert radiation into “color”?
• What part of the spectrum do we see?

​

Visible light

We “see” electromagnetic radiation in a range of wavelengths

Light spectrum

• The appearance of light depends on its power spectrum
• How much power (or energy) at each wavelength

daylight

tungsten bulb

Our visual system converts a light spectrum into “color”

• This is a rather complex transformation

The human visual system

• Color perception
• Light hits the retina, which contains photosensitive cells
• rods and cones
• These cells convert the spectrum into a few discrete values

Density of rods and cones

• Rods and cones are non-uniformly distributed on the retina
• Rods responsible for intensity, cones responsible for color
• Fovea: Small region (1 or 2°) at the center of the visual field containing the highest density of cones (and no rods).
• Less visual acuity in the periphery—many rods wired to the same neuron

Demonstrations of visual acuity

With one eye shut, at the right distance, all of these letters should appear equally legible (Glassner, 1.7).

Demonstrations of visual acuity

With left eye shut, look at the cross on the left. At the right distance, the circle on the right should disappear (Glassner, 1.8).

Brightness contrast and constancy

• The apparent brightness depends on the surrounding region
• brightness contrast: a constant colored region seems lighter or darker depending on the surrounding intensity

​

​

​

​

​

• brightness constancy: a surface looks the same under widely varying lighting conditions.

Light response is nonlinear

• Our visual system has a large dynamic range
• We can resolve both light and dark things at the same time
• One mechanism for achieving this is that we sense light intensity on a logarithmic scale
• an exponential intensity ramp will be seen as a linear ramp
• Another mechanism is adaptation
• rods and cones adapt to be more sensitive in low light, less sensitive in bright light.

Visual dynamic range

A piece of white paper can be 1,000,000,000 times brighter in outdoor sunlight than in a moonless night.

BUT in a given lighting condition, light perception ranges over only about two orders of magnitude.

Visual dynamic range

A piece of white paper can be 1,000,000,000 times brighter in outdoor sunlight than in a moonless night.

BUT in a given lighting condition, light perception ranges over only about two orders of magnitude.

https://threader.app/thread/1134003178515701762

https://threader.app/thread/1134003178515701762

https://en.wikipedia.org/wiki/Daylight

http://cchen156.web.engr.illinois.edu/SID.html

Visual dynamic range

Dark night

Indoor lighting

Cloudy day

Sunny day

If we were sensitive to this whole range all the time, we wouldn’t be able to

discriminate lightness levels in a typical scene.

The visual system solves this problem by restricting the ‘dynamic range’ of its

response to match the current overall or ‘ambient’ light level.

Color perception

• Three types of cones
• Each is sensitive in a different region of the spectrum
• but regions overlap
• Short (S) corresponds to blue
• Medium (M) corresponds to green
• Long (L) corresponds to red
• Different sensitivities: we are more sensitive to green than red
• varies from person to person (and with age)
• Colorblindness—deficiency in at least one type of cone

Color perception

• Rods and cones act as filters on the spectrum
• To get the output of a filter, multiply its response curve by the spectrum, integrate over all wavelengths
• Each cone yields one number
• Q: How can we represent an entire spectrum with 3 numbers?
• A: We can’t! Most of the information is lost
• As a result, two different spectra may appear indistinguishable
• such spectra are known as metamers

Wavelength

Power

Metamers

1. Combined light spectrum

Metamers

1. Combined light spectrum

2. Cone sensitivity (S, M, L)

3. Multiplication of 1 and 2

Metamers

1. Combined light spectrum

2. Cone sensitivity (S, M, L)

3. Multiplication of 1 and 2

4. Observed color (yellow)

What kind of bulb is it?

http://www.chemistryland.com/CHM107Lab/Exp7/Spectroscope/Spectroscope.html

What color is the dress?

• White and gold?
• Black and blue?

What color is the dress?

ⓘ Start presenting to display the poll results on this slide.

Reflectance and Illumination In Popular Culture…

What color is the center ball?

What color is the center ball?

Perception summary

• The mapping from radiance to perceived color is quite complex!
• We throw away most of the data
• We apply a logarithm
• Brightness affected by pupil size and adaptation of rods/cones
• Brightness contrast and constancy effects

​

​

• The same is true for cameras
• But we have tools to correct for these effects
• (Computational Photography)

Cameras also see color

• Common technique is to place a mosaic of color filters (a Bayer filter) in front of the sensor

​

• Colors are interpolated to create a full-resolution “demosaicked” color image

Bayer filter pattern in front of sensor

https://en.wikipedia.org/wiki/Bayer_filter

Early color photography

• Prior to the invention of color film, Sergey Prokudin-Gorsky took three separate exposures with three different color filters

Film has its own sensitivity

• “… the film of Lincoln’s era was sensitive only to blue and UV light, causing cheeks to appear dark, and overly emphasizing wrinkles by filtering out skin subsurface scatter which occurs mostly in the red channel. Hence, the deep lines and sharp creases that we associate with Lincoln’s face are likely exaggerated by the photographic process of the time.”

Time-Travel Rephotography

Xuan Luo, Xuaner Zhang, Paul Yoo, Ricardo Martin-Brualla, Jason Lawrence, Steven M. Seitz

SIGGRAPH Asia 2021

Questions?