Nima Kalantari
CSCE 448/748 – Computational Photography
Sampling, Frequency, and Filtering
Many slides from Alexei A. Efros, James Hayes, Rob Fergus
Outline
Image formation involves sampling
Image formation involves sampling
Scene
Digital image
Aliasing in images (Moire pattern)
Original
Aliased
Credit: Scientific Volume Imaging
Aliasing in video (wagon-wheel effect)
Created by Jesse Mason, https://www.youtube.com/watch?v=QOwzkND_ooU
Aliasing in video (wagon wheel effect)
Slide by Steve Seitz
Aliasing in video (wagon wheel effect)
Sampling Artifacts in Computer Graphics
Antialiasing
Video: Point vs Antialiased Sampling
Point in Time
Motion Blurred
Video: Point Sampling in Time
30 fps video. 1/800 second exposure is sharp in time, causes time aliasing.
Credit: Aris & cams youtube, https://youtu.be/NoWwxTktoFs
Imaging: sampling in space
Note jagged edges of the flower petals
Sample
Imaging: antialiased sampling
Pre-Filter
Sample
Note antialiased edges
Aliasing and Antialiasing
Outline
Image representation with basis functions
Spatial basis (1D example)
1
2
3
1
2
3
4
1
2
3
1
2
3
4
1
2
3
1
2
3
4
1
2
3
1
2
3
4
B1
B2
B3
F
F = 4 B1 + 1 B2 + 2 B3
Spatial basis
Frequency domain
Sines and Cosines
Frequencies
Fourier Transform
of sines and cosines
Joseph Fourier 1768 - 1830
f(target) = A0f0 + A1f1 + A2f2 + …
Extension to 2D
Image as a sum of basis images
=
Constant
Frequency Domain
(0,0)
Spatial Domain
— frequency 1/32; 32 pixels per cycle
Max signal freq =1/32
(0,0)
Frequency Domain
Spatial Domain
— frequency 1/16; 16 pixels per cycle
Max signal freq =1/16
(0,0)
Frequency Domain
Spatial Domain
Frequency Domain
Spatial Domain
2D Frequency Space
Note: Frequency domain also known as frequency space, Fourier domain, spectrum, …
Frequency Domain
Spatial Domain
Sampling
© 2006 Steve Marschner
Undersampling
© 2006 Steve Marschner
Undersampling
© 2006 Steve Marschner
Aliasing in images
Original
Aliased
Credit: Scientific Volume Imaging
Moiré Patterns in Imaging
lystit.com
Read every sensor pixel
Skip odd rows and columns
Higher Frequencies Need Faster Sampling
x
f1(x)
f2(x)
f3(x)
f4(x)
f5(x)
f2(x)
f1(x)
f3(x)
f4(x)
f5(x)
Periodic sampling locations
Low-frequency signal: sampled adequately for reasonable reconstruction
High-frequency signal is insufficiently sampled: reconstruction incorrectly appears to be from a low frequency signal
Nyquist Theorem
Signal Interval
Maximum Acceptable
Sampling Interval
Nyquist Theorem
Nyquist Theorem
Nyquist Theorem
Antialiasing
Outline
Moving Average
Weighted Moving Average
/5
Weighted Moving Average
/5
/16
Weighted Moving Average In 2D
© 2006 Steve Marschner • 45
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 0 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| | |
| | |
| | |
Slide by Steve Seitz
Cross-correlation
Moving Average In 2D
© 2006 Steve Marschner • 47
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 0 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| | |
| | |
| | |
Slide by Steve Seitz
Image filtering
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 0 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| | | | | | | | | |
| 0 | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
Credit: S. Seitz
1
1
1
1
1
1
1
1
1
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 0 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| | | | | | | | | |
| 0 | 10 | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
Image filtering
1
1
1
1
1
1
1
1
1
Credit: S. Seitz
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 0 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| | | | | | | | | |
| 0 | 10 | 20 | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
Image filtering
1
1
1
1
1
1
1
1
1
Credit: S. Seitz
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 0 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| | | | | | | | | |
| 0 | 10 | 20 | 30 | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
Image filtering
1
1
1
1
1
1
1
1
1
Credit: S. Seitz
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 0 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| | | | | | | | | |
| 0 | 10 | 20 | 30 | 30 | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
Image filtering
1
1
1
1
1
1
1
1
1
Credit: S. Seitz
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 0 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| | | | | | | | | |
| 0 | 10 | 20 | 30 | 30 | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | 50 | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
Image filtering
1
1
1
1
1
1
1
1
1
Credit: S. Seitz
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 0 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| | | | | | | | | |
| 0 | 10 | 20 | 30 | 30 | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | 90 | | | |
| | | | | | | | | |
| | | | 50 | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
Image filtering
1
1
1
1
1
1
1
1
1
Credit: S. Seitz
| | | | | | | | | |
| 0 | 10 | 20 | 30 | 30 | 30 | 20 | 10 | |
| 0 | 20 | 40 | 60 | 60 | 60 | 40 | 20 | |
| 0 | 30 | 60 | 90 | 90 | 90 | 60 | 30 | |
| 0 | 30 | 50 | 80 | 80 | 90 | 60 | 30 | |
| 0 | 30 | 50 | 80 | 80 | 90 | 60 | 30 | |
| 0 | 20 | 30 | 50 | 50 | 60 | 40 | 20 | |
| 10 | 20 | 30 | 30 | 30 | 30 | 20 | 10 | |
| 10 | 10 | 10 | 0 | 0 | 0 | 0 | 0 | |
| | | | | | | | | |
Image filtering
1
1
1
1
1
1
1
1
1
Credit: S. Seitz
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 0 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Box filter
1
1
1
1
1
1
1
1
1
Slide credit: David Lowe (UBC)
Cross-correlation vs. Convolution
Slide by Steve Seitz
Convolution
Adapted from F. Durand
Convolution is nice!
Convolution is nice!
Gaussian filtering
1 | 2 | 1 |
2 | 4 | 2 |
1 | 2 | 1 |
Slide by Steve Seitz
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 0 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 90 | 90 | 90 | 90 | 90 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Gaussian filter
0.003 0.013 0.022 0.013 0.003
0.013 0.059 0.097 0.059 0.013
0.022 0.097 0.159 0.097 0.022
0.013 0.059 0.097 0.059 0.013
0.003 0.013 0.022 0.013 0.003
5 x 5, σ = 1
Slide credit: Christopher Rasmussen
Gaussian Kernel
Source: K. Grauman
σ = 2 with 30 x 30 kernel
σ = 5 with 30 x 30 kernel
Gaussian filters
= 30 pixels
= 1 pixel
= 5 pixels
= 10 pixels
Choosing kernel width
Source: K. Grauman
Practical matters
Side by Derek Hoiem
Gaussian and convolution
Source: K. Grauman
*
=
Box vs. Gaussian filtering
Slide by Steve Seitz
Filtering in Frequency Domain
Visualizing Image Frequency Content
Spatial Domain
Frequency Domain
Filtering in the Frequency Domain
Spatial Domain
Frequency Domain
Fourier
Process
Inverse
Fourier
Spatial Domain
Frequency Domain
Filter Out Low Frequencies Only (Edges)
Spatial Domain
Frequency Domain
Filter Out High Frequencies (Blur)
Spatial Domain
Frequency Domain
Filter Out Low and High Frequencies
Spatial Domain
Frequency Domain
Filter Out Low and High Frequencies
Spatial Domain
Frequency Domain
Relationship between the two
The Convolution Theorem
2D convolution theorem example
*
f(x,y)
h(x,y)
g(x,y)
|F(sx,sy)|
|H(sx,sy)|
|G(sx,sy)|
Box vs. Gaussian filtering
Gaussian
Box filter
Gaussian filter
Spatial Domain
Frequency Domain
Box filter
Spatial Domain
Frequency Domain
Outline
Image formation involves sampling
Antialiasing
Digital sensors
Pixel area
Light sensitive area
small to large fill factor
Fill factor effect
4%
100%
Outline
Linear filters: examples
Original
1
1
1
1
1
1
1
1
1
Blur
(with a mean filter)
Source: D. Lowe
=
Practice with linear filters
0
0
0
0
1
0
0
0
0
Original
?
Source: D. Lowe
Practice with linear filters
0
0
0
0
1
0
0
0
0
Original
Filtered
(no change)
Source: D. Lowe
Practice with linear filters
0
0
0
1
0
0
0
0
0
Original
?
Source: D. Lowe
Practice with linear filters
0
0
0
1
0
0
0
0
0
Original
Shifted left
By 1 pixel
Source: D. Lowe
Linear filters: examples
Original
1
1
1
1
1
1
1
1
1
0
0
0
0
2
0
0
0
0
-
Sharpening filter (accentuates edges)
Source: D. Lowe
=
Filtering – sharpening
-
Image
Smoothed
=
Details
Filtering – Sharpening
+α
Image
Details
=
“Sharpened” α=1
Filtering – Sharpening
=
+α
Image
Details
“Sharpened” α=0
Filtering – Sharpening
=
+α
Image
Details
“Sharpened” α=2
Filtering – Sharpening
=
+α
Image
Details
“Sharpened” α=0
Filtering – Extreme Sharpening
=
+α
Image
Details
“Sharpened” α=10
Unsharp mask filter
Gaussian
unit impulse
Laplacian of Gaussian
image
blurred�image
unit impulse�(identity)
Edges
Edges are caused by a variety of factors
depth discontinuity
surface color discontinuity
illumination discontinuity
surface normal discontinuity
Origin of edges
Credit: Ali Farhadi
Characterizing edges
image
intensity function�(along horizontal scanline)
first derivative
edges correspond to�extrema of derivative
Credit: Ali Farhadi
Partial derivatives with convolution
Source: K. Grauman
-1 1
-1 1
x
y
Partial derivatives of an image
Which shows changes with respect to x?
-1 1
-1 1
Finite difference filters
-1 | 0 | 1 |
-1 | 0 | 1 |
-1 | 0 | 1 |
1 | 1 | 1 |
0 | 0 | 0 |
-1 | -1 | -1 |
-1 | 0 | 1 |
-2 | 0 | 2 |
-1 | 0 | 1 |
1 | 2 | 1 |
0 | 0 | 0 |
-1 | -2 | -1 |
1 | 0 |
0 | -1 |
0 | 1 |
-1 | 0 |
Prewitt
Sobel
Roberts
Image gradient
Source: Steve Seitz
Image Gradient
Effects of noise
Source: S. Seitz
Solution: smooth first
f
h
f * h
Source: S. Seitz
Noise in 2D
Noisy Input
dx via [-1,1]
Zoom
Source: D. Fouhey
Noise + Smoothing
Smoothed Input
dx via [-1,1]
Zoom
Source: D. Fouhey
Derivative theorem of convolution
Derivative of Gaussian filter
* [1 -1] =
Derivative of Gaussian filter
x-direction
y-direction
Vertical
Horizontal
Practical matters
f
g
g
g
g
f
g
g
g
g
f
g
g
g
g
full
same
valid
Source: S. Lazebnik
Practical matters
Source: S. Marschner
Summary
A cool application
Hybrid images
Hybrid Images
Gaussian
unit impulse
Laplacian of Gaussian
A. Oliva, A. Torralba, P.G. Schyns, “Hybrid Images,” SIGGRAPH 2006
Gaussian
Laplacian
Da Vinci and Peripheral Vision
Da Vinci and Peripheral Vision