Nima Kalantari
CSCE 448/748 - Computational Photography
Coded Exposure and Aperture
Many Slides from Rob Fergus, Raskar et al., and Levin et al.
Different types of blur
Different types of blur
Let’s take a photo
Blurry result
Slow-motion replay
Slow-motion replay
Motion of camera
Image formation process
=
⊗
Blurry image
Sharp image
Blur �kernel
Convolution�operator
Different types of blur
Different types of blur
Blurring
==
Convolution
Sync Function
Different types of blur
Different types of blur
Lens and defocus
Lens
Camera sensor
Point spread function
Focal plane
Lens and defocus
Lens
Object
Camera sensor
Point spread function
Focal plane
Lens and defocus
Lens
Camera sensor
Point spread function
Object
Focal plane
Lens and defocus
Lens
Camera sensor
Point spread function
Object
Focal plane
Lens and defocus
Lens
Camera sensor
Point spread function
Object
Focal plane
DoF & aperture
f/2.8
f/32
Lens and defocus
Lens
Camera sensor
Point spread function
Image of a point light source
Focal plane
�Lens’ aperture �
Lens and defocus
Lens
Object
Camera sensor
Point spread function
Image of a defocused point light source
�Lens’ aperture �
Focal plane
Lens and defocus
Lens
Camera sensor
Point spread function
Image of a defocused point light source
Object
�Lens’ aperture �
Focal plane
Lens and defocus
Lens
Camera sensor
Point spread function
Image of a defocused point light source
�Lens’ aperture �
Object
Focal plane
Lens and defocus
Lens
Camera sensor
Point spread function
Image of a defocused point light source
�Lens’ aperture �
Object
Focal plane
Blurring
==
Convolution
Sync Function
Blurring
==
Convolution
Traditional Camera: Box Filter
Sync Function
Coded exposure (flutter shutter)
Traditional Camera
Shutter is OPEN
Our Camera
Flutter Shutter
Shutter is OPEN and CLOSED
Comparison of Blurred Images
Implementation
Completely Portable
Lab Setup
Blurring
==
Convolution
Traditional Camera: Box Filter
Sync Function
Flutter Shutter: Coded Filter
Preserves High Frequencies!!!
Comparison
Inverse Filter Unstable
Inverse Filter stable
Short Exposure
Long Exposure
Coded Exposure
Ground Truth
Matlab Lucy
Our result
Are all codes “good”?
Alternate
All ones
Random
Our Code
License Plate Retrieval
License Plate Retrieval
Removing Camera Shake
Removing Camera Shake
Image and Depth from a Conventional Camera with a Coded Aperture�
Anat Levin, Rob Fergus, Frédo Durand, William Freeman
MIT CSAIL
Single input image:
Output #1: Depth map
Single input image:
Output #1: Depth map
Output #2: All-focused image
Lens and defocus
Lens
Camera sensor
Point spread function
Image of a point light source
Focal plane
�Lens’ aperture �
Lens and defocus
Lens
Object
Camera sensor
Point spread function
Image of a defocused point light source
�Lens’ aperture �
Focal plane
Lens and defocus
Lens
Camera sensor
Point spread function
Image of a defocused point light source
Object
�Lens’ aperture �
Focal plane
Lens and defocus
Lens
Camera sensor
Point spread function
Image of a defocused point light source
�Lens’ aperture �
Object
Focal plane
Lens and defocus
Lens
Camera sensor
Point spread function
Image of a defocused point light source
�Lens’ aperture �
Object
Focal plane
Depth and defocus
Depth from defocus:
Infer depth by analyzing local scale of defocus blur
Out of focus
In focus
ill posed
Challenges
Input
Ringing with conventional deblurring algorithm
Out of focus
?
Outline
Outline
?
=
Deconvolution is ill posed
Deconvolution is ill posed
?
=
=
?
Solution 1:
Solution 2:
Idea 1: Natural images prior
Image
gradient
put a penalty on gradients
Natural images have sparse gradients
Natural
Unnatural
What makes images special?
Deconvolution with prior
_
2
+
_
+
2
?
?
Convolution error
Derivatives prior
High
Low
Equal convolution error
Comparing deconvolution algorithms
Input
Richardson-Lucy
(Non blind) deconvolution code available online: http://groups.csail.mit.edu/graphics/CodedAperture/
Gaussian prior
“spread” gradients
Sparse prior
“localizes” gradients
Comparing deconvolution algorithms
Input
Richardson-Lucy
(Non blind) deconvolution code available online: http://groups.csail.mit.edu/graphics/CodedAperture/
Gaussian prior
“spread” gradients
Sparse prior
“localizes” gradients
Outline
Outline
DoF & aperture
f/2.8
f/32
Defocus as local convolution
Input defocused image
Calibrated blur kernels at different depths
Defocus as local convolution
Depth k=1:
Depth k=2:
Depth k=3:
Input defocused image
Local �sub-window
Calibrated blur kernels at depth
Sharp �sub-window
Overview
Correct scale
Smaller scale
Larger scale
Try deconvolving local input windows with different scaled filters:
Somehow: select best scale.
?
?
?
Challenge
?
?
Correct scale
Smaller scale
?
Larger scale
Idea 2: Coded Aperture
- make defocus patterns different from natural images and easier to discriminate
Conventional �aperture
Our coded aperture
Solution: lens with occluder
Lens
Camera sensor
Point spread function
Object
Focal plane
Solution: lens with occluder
Lens with coded aperture
Camera sensor
Point spread function
Image of a defocused point light source
�Aperture pattern�
Object
Focal plane
Solution: lens with occluder
Lens with coded aperture
Camera sensor
Point spread function
Image of a defocused point light source
�Aperture pattern�
Object
Focal plane
Solution: lens with occluder
Lens with coded aperture
Camera sensor
Point spread function
Image of a defocused point light source
�Aperture pattern�
Object
Focal plane
Solution: lens with occluder
Lens with coded aperture
Camera sensor
Point spread function
Image of a defocused point light source
�Aperture pattern�
Object
Focal plane
Solution: lens with occluder
Lens with coded aperture
Camera sensor
Point spread function
Image of a defocused point light source
�Aperture pattern�
Object
Focal plane
Why coded?
Conventional
Coded
Coded aperture- reduce uncertainty in scale identification
Correct scale
Smaller scale
Larger scale
Filter Design
Sampled aperture patterns
Conventional aperture
More discrimination between scales
Score
Less discrimination between scales
See paper for details
Analytically search for a pattern maximizing discrimination between images at different defocus scales (KL-divergence)
Account for image prior and physical constraints
Depth results
Input
Local depth estimation
Regularized depth
Regularizing depth estimation
_
2
+
Convolution error
Derivatives prior
Try deblurring with 10 different aperture scales
Keep minimal error scale in each local window
+ regularization
Input
Local depth estimation
Regularized depth
Regularizing depth estimation
Sometimes, manual intervention
Input
After user corrections
Regularized depth
Local depth estimation
All focused results
Input
All-focused (deconvolved)
Close-up
Original image
All-focus image
Input
All-focused (deconvolved)
Close-up
Original image
All-focus image
Naïve sharpening
Comparison- conventional aperture result
Ringing due to wrong scale estimation
Comparison- coded aperture result
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image