What food are you craving?

ⓘ Start presenting to display the poll results on this slide.

Reflectance & Photometric stereo

CS5670 : Computer Vision

Reading

• Szeliski 2^nd Edition: Chapter 2.2 & 13.1

Announcements

• Project 4 (Stereo) released today due Friday, March 31, at 8pm
• To be done in groups of 2

​

Roadmap for the rest of the course

• The next three lectures will finish up geometry and image formation
• Next up (after Spring Break): deep learning, image recognition, neural radiance fields, image generation models
• Coming up
• Reflectance and Photometric Stereo (today)
• Two-view geometry
• Multi-view geometry

Can we determine shape from lighting?

• Are these spheres?
• Or just flat discs painted with varying albedo?

A Single Image: Shape from Shading

Assume is 1 for now.

What can we measure from one image?

• is the angle between N and L
• Add assumptions:
• Constant albedo
• A few known normals (e.g. silhouettes)
• Smoothness of normals

In practice, SFS doesn’t work very well:

assumptions are too restrictive,

too much ambiguity in nontrivial scenes.

Reflectance

Light sources

• Basic types
• point source
• directional source
• a point source that is infinitely far away
• area source
• a union of point sources

​

• What happens when light hits an object?

Modeling Image Formation

We need to reason about:

• How light interacts with the scene
• How a pixel value is related to light energy in the world

Directional Lighting

• Key property: all rays are parallel
• Equivalent to an infinitely distant point source

Lambertian Reflectance

Materials - Three Forms

© Kavita Bala, Computer Science, Cornell University

Reflectance—Three Forms

© Kavita Bala, Computer Science, Cornell University

Ideal diffuse (Lambertian)

Directional

diffuse

Ideal

specular

Ideal Diffuse Reflection

• Characteristic of multiple scattering materials
• An idealization but reasonable for matte surfaces

​

© Kavita Bala, Computer Science, Cornell University

Lambertian Reflectance

1. Reflected energy is proportional to cosine of angle between L and N (incoming)

​

• Measured intensity is viewpoint-independent (outgoing)

Lambertian Reflectance: Incoming

• Reflected energy is proportional to cosine of angle between L and N

​

Lambertian Reflectance: Incoming

• Reflected energy is proportional to cosine of angle between L and N

​

Lambertian Reflectance: Incoming

• Reflected energy is proportional to cosine of angle between L and N

​

Light hitting surface is proportional to the cosine

Lambertian Reflectance: Outgoing

• Radiance (what the eye sees) is viewpoint-independent

Lambertian Reflectance: Outgoing

• Radiance (what the eye sees) is viewpoint-independent

​

Lambertian Reflectance: Outgoing

• Radiance (what the eye sees) is viewpoint-independent

Lambertian Reflectance: Outgoing

• Radiance (what the eye sees) is viewpoint-independent

Radiance

(what eye sees)

A cos (θ)

Lambertian appearance is view-independent

• Number of photons reflected to a given angle θ is proportional to cos(θ)

Lambertian appearance is view-independent

• But appearance is the same from every angle due to larger pixel footprint at larger angles

Lambert's cosine law:

• Number of photons reflected to a given angle θ is proportional to cos(θ)

Lambertian appearance is view-independent

• But appearance is the same from every angle due to larger pixel footprint at larger angles

Lambert's cosine law:

• Number of photons reflected to a given angle θ is proportional to cos(θ)

Lambertian Surfaces: Appearance vs Reflected Photons

• Appearance is the same from every angle

• Radiant Intensity? (how many photons reflected per angle)

Lambertian Surfaces: Appearance vs Reflected Photons

• Appearance is the same from every angle

• Radiant Intensity? (how many photons reflected per angle)

Lambertian Surfaces: Appearance vs Reflected Photons

• Appearance is the same from every angle

• Radiant Intensity? (how many photons reflected per angle)

Lambertian appearance is view-independent

• But appearance is the same from every angle due to larger pixel footprint at larger angles

Lambert's cosine law:

• Number of photons reflected to a given angle θ is proportional to cos(θ)

Final Lambertian image formation model

1. Diffuse albedo: what fraction of incoming light is reflected?
• Introduce scale factor
2. Light intensity: how much light is arriving?
• Compensate with camera exposure (global scale factor)
3. Camera response function
• Assume pixel value is linearly proportional to incoming energy (perform radiometric calibration if not)

Albedo

Source: https://en.wikipedia.org/wiki/Albedo

A Single Image: Shape from Shading

Assume is 1 (for now)

What can we measure from one image?

• is the angle between N and L
• Add assumptions:
• Constant albedo
• A few known normals (e.g., silhouettes)
• Smoothness of normals

In practice, SFS doesn’t work well (yet):

assumptions are too restrictive,

too much ambiguity in nontrivial scenes.

A Single Image: Shape from shading

Suppose (for now)

You can directly measure angle between normal and light source

• Not quite enough information to compute surface shape
• But can be if you add some additional info, for example
• assume a few of the normals are known (e.g., along silhouette)
• constraints on neighboring normals—“integrability”
• smoothness
• Hard to get it to work well in practice
• plus, how many real objects have constant albedo?
• But, deep learning can help

https://www.good.is/optical-illusion-plates-and-bowls-upside-down-or-not

Application: Detecting composite photos

Fake photo

Real photo

A Single Image: Shape from shading

Suppose (for now)

You can directly measure angle between normal and light source

• Not quite enough information to compute surface shape
• But can be if you add some additional info, for example
• assume a few of the normals are known (e.g., along silhouette)
• constraints on neighboring normals—“integrability”
• smoothness
• Hard to get it to work well in practice
• plus, how many real objects have constant albedo?
• But, deep learning can help

Diffuse reflection

http://www.math.montana.edu/frankw/ccp/multiworld/twothree/lighting/applet1.htm

http://www.math.montana.edu/frankw/ccp/multiworld/twothree/lighting/learn2.htm

​

Demo

Let’s take more than one photo!

Photometric stereo

N

V

Can write this as a matrix equation:

Solving the equations

Solve one such linear system per pixel to solve for that pixel’s surface normal

More than three lights

Can get better results by using more than 3 lights

Computing light source directions

Trick: place a chrome sphere in the scene

​

​

​

​

​

​

• the location of the highlight tells you where the light source is

Recall the rule for specular reflection

For a perfect mirror, light is reflected about N

We see a highlight when V = R

• then L is given as follows:

Example

Forsyth & Ponce, Sec. 5.4

Input views

Depth from normals

• Solving the linear system per-pixel gives us an estimated surface normal for each pixel

​

• How can we compute depth from normals?
• Normals are like the “derivative” of the true depth

Input photo

Estimated normals

Estimated normals (needle diagram)

Normal Integration

• Integrating a set of derivatives is easy in 1D
• (similar to Euler’s method from diff. eq. class)

​

• Could integrate normals in each column / row separately
• Wouldn’t give a good surface
• Instead, we formulate as a linear system and solve for depths that best agree with the surface normals

Depth from normals

Get a similar equation for V₂

• Each normal gives us two linear constraints on z
• compute z values by solving a matrix equation

Results

from Athos Georghiades

Results

Extension

• Photometric Stereo from Colored Lighting

Video Normals from Colored Lights

Gabriel J. Brostow, Carlos Hernández, George Vogiatzis, Björn Stenger, Roberto Cipolla

IEEE TPAMI, Vol. 33, No. 10, pages 2104-2114, October 2011.

Questions?

For now, ignore specular reflection

Slides from Photometric Methods for 3D Modeling, Matsushita, Wilburn, Ben-Ezra

And Refraction…

Slides from Photometric Methods for 3D Modeling, Matsushita, Wilburn, Ben-Ezra

And Interreflections…

Slides from Photometric Methods for 3D Modeling, Matsushita, Wilburn, Ben-Ezra

And Subsurface Scattering…

Slides from Photometric Methods for 3D Modeling, Matsushita, Wilburn, Ben-Ezra

Limitations

Bigger problems

• doesn’t work for shiny things, semi-translucent things
• shadows, inter-reflections

Smaller problems

• camera and lights have to be distant
• calibration requirements
• measure light source directions, intensities
• camera response function

Newer work addresses some of these issues

Some pointers for further reading:

• Zickler, Belhumeur, and Kriegman, "Helmholtz Stereopsis: Exploiting Reciprocity for Surface Reconstruction." IJCV, Vol. 49 No. 2/3, pp 215-227.
• Hertzmann & Seitz, “Example-Based Photometric Stereo: Shape Reconstruction with General, Varying BRDFs.” IEEE Trans. PAMI 2005

Johnson and Adelson, 2009

Johnson and Adelson, 2009

https://www.youtube.com/watch?v=S7gXih4XS7A

Questions?