2 of 89

Logistics:

Homework 0 is due tonight!
Homework 1 will come out later today

4 of 89

Resize 4x4 -> 7x7

Create our new image
Match up coordinates

5 of 89

Resize 4x4 -> 7x7

Create our new image
Match up coordinates

4/7 X - 3/14 = Y

Iterate over new pts

Map to old coords

6 of 89

Resize 4x4 -> 7x7

Create our new image
Match up coordinates

4/7 X - 3/14 = Y

Iterate over new pts

Map to old coords
(1, 3) -> (5/14, 7/14)
Interpolate old values

q = (166, 172.5, 100)

7 of 89

Resize 4x4 -> 7x7

Create our new image
Match up coordinates

4/7 X - 3/14 = Y

Iterate over new pts

Map to old coords
(1, 3) -> (5/14, 7/14)
Interpolate old values

q = (166, 172.5, 100)

Fill in the rest

8 of 89

Different methods

9 of 89

For shrinking interpolation bad, averaging good

“interpolation”

averaging

10 of 89

Convolution: Weighted sum over pixels

13 of 89

Cross-Correlation vs Convolution

14 of 89

Cross-Correlation vs Convolution

Do this in HW!

15 of 89

Cross-Correlation vs Convolution

These are nicer mathematically

16 of 89

So what can we do with these convolutions anyway?

Mathematically: all the nice things

Commutative

A*B = B*A

Associative

A*(B*C) = (A*B)*C

Distributes over addition

A*(B+C) = A*B + A*C

Plays well with scalars

x(A*B) = (xA)*B = A*(xB)

17 of 89

So what can we do with these convolutions anyway?

This means some convolutions decompose:

2d gaussian is just composition of 1d gaussians

Faster to run 2 1d convolutions

18 of 89

So what can we do with these convolutions anyway?

Blurring
Sharpening
Edges
Features
Derivatives
Super-resolution
Classification
Detection
Image captioning
...

19 of 89

So what can we do with these convolutions anyway?

Blurring
Sharpening
Edges
Features
Derivatives
Super-resolution
Classification
Detection
Image captioning
...

All of computer vision

is convolutions

(basically)

20 of 89

So what can we do with these convolutions anyway?

Blurring
Sharpening
Edges
Features
Derivatives
Super-resolution
Classification
Detection
Image captioning
...

All of computer vision

is convolutions

(basically)

21 of 89

What’s an edge?

Image is a function
Edges are rapid changes in this function

22 of 89

What’s an edge?

Image is a function
Edges are rapid changes in this function

23 of 89

Finding edges

Could take derivative
Edges = high response

24 of 89

Image derivatives

Recall:

We don’t have an “actual”�Function, must estimate
Possibility: set h = 1
What will that look like?

25 of 89

Image derivatives

Recall:

We don’t have an “actual”�Function, must estimate
Possibility: set h = 1
What will that look like?

26 of 89

Image derivatives

Recall:

We don’t have an “actual”�Function, must estimate
Possibility: set h = 2
What will that look like?

27 of 89

Image derivatives

Recall:

We don’t have an “actual”�Function, must estimate
Possibility: set h = 2
What will that look like?

28 of 89

Images are noisy!

29 of 89

But we already know how to smooth

* =

30 of 89

But we already know how to smooth

* =

31 of 89

Smooth first, then derivative

32 of 89

Smooth first, then derivative

33 of 89

Smooth first, then derivative

34 of 89

Smooth first, then derivative

35 of 89

Smooth first, then derivative

36 of 89

Smooth first, then derivative

37 of 89

Smooth first, then derivative

38 of 89

Smooth first, then derivative

39 of 89

Smooth first, then derivative

40 of 89

Smooth first, then derivative

41 of 89

Smooth first, then derivative

42 of 89

Sobel filter! Smooth & derivative

43 of 89

Image derivatives

Recall:

Want smoothing too!

44 of 89

Finding edges

Could take derivative
Find high responses
Sobel filters!
But…

45 of 89

Finding edges

Could take derivative
Find high responses
Sobel filters!
But…
Edges go both ways
Want to find extrema

46 of 89

2nd derivative!

Crosses zero at extrema

47 of 89

Laplacian (2nd derivative)!

Crosses zero at extrema
Recall:

Laplacian:

Again, have to�estimate f’’(x):

48 of 89

Laplacians

Laplacian:

49 of 89

Laplacians

Laplacian:

Measures the divergence of the gradient

Flux of gradient vector field through small area

56 of 89

Laplacians

Laplacian:

57 of 89

Laplacians

Laplacian:

58 of 89

Laplacians

Laplacian:

59 of 89

Laplacians

Laplacian:

60 of 89

Laplacians

Laplacian:

Negative Laplacian, -4 in middle
Positive Laplacian --->

61 of 89

Laplacians also sensitive to noise

Again, use gaussian smoothing
Can just use one kernel since convs commute
Laplacian of Gaussian, LoG
Can get good approx. with�5x5 - 9x9 kernels

62 of 89

Another edge detector:

Image is a function:

Has high frequency and low frequency components
Think in terms of fourier transform

Edges are high frequency changes
Maybe we want to find edges of a specific size (i.e. specific frequency)

63 of 89

Difference of Gaussian (DoG)

Gaussian is a low pass filter
Strongly reduce components with frequency f < σ
(g*I) low frequency components
I - (g*I) high frequency components
g(σ1)*I - g(σ2)*I

Components in between these frequencies

g(σ1)*I - g(σ2)*I = [g(σ1) - g(σ2)]*I

64 of 89

Difference of Gaussian (DoG)

g(σ1)*I - g(σ2)*I = [g(σ1) - g(σ2)]*I

σ = 2

σ = 1

65 of 89

Difference of Gaussian (DoG)

g(σ1)*I - g(σ2)*I = [g(σ1) - g(σ2)]*I
This looks a lot like our LoG!
(not actually the same but similar)

66 of 89

DoG (1 - 0)

67 of 89

DoG (2 - 1)

68 of 89

DoG (3 - 2)

69 of 89

DoG (4 - 3)

70 of 89

Another approach: gradient magnitude

Don’t need 2nd derivatives
Just use magnitude of gradient

71 of 89

Another approach: gradient magnitude

Don’t need 2nd derivatives
Just use magnitude of gradient
Are we done? No!

72 of 89

Another approach: gradient magnitude

Don’t need 2nd derivatives
Just use magnitude of gradient
Are we done? No!

73 of 89

What we really want: line drawing

74 of 89

Canny Edge Detection

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/

Your first image processing pipeline!

Old-school CV is all about pipelines

Algorithm:

Smooth image (only want “real” edges, not noise)
Calculate gradient direction and magnitude
Non-maximum suppression perpendicular to edge
Threshold into strong, weak, no edge
Connect together components

75 of 89

Smooth image

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/

You know how to do this, gaussians!

76 of 89

Gradient magnitude and direction

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/

Sobel filter

77 of 89

Non-maximum suppression

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/

Want single pixel edges, not thick blurry lines
Need to check nearby pixels
See if response is highest

78 of 89

Non-maximum suppression

79 of 89

Non-maximum suppression

80 of 89

Non-maximum suppression

81 of 89

Non-maximum suppression

82 of 89

Non-maximum suppression

83 of 89

Non-maximum suppression

84 of 89

Non-maximum suppression

85 of 89

Non-maximum suppression

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/

86 of 89

Threshold edges

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/

Still some noise
Only want strong edges
2 thresholds, 3 cases

R > T: strong edge
R < T but R > t: weak edge
R < t: no edge

Why two thresholds?

87 of 89

Connect ‘em up!

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/

Strong edges are edges!
Weak edges are edges �iff they connect to strong
Look in some neighborhood�(usually 8 closest)

88 of 89

Canny Edge Detection

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/

Your first image processing pipeline!

Old-school CV is all about pipelines

Algorithm:

Smooth image (only want “real” edges, not noise)
Calculate gradient direction and magnitude
Non-maximum suppression perpendicular to edge
Threshold into strong, weak, no edge
Connect together components
Tunable: Sigma, thresholds

89 of 89

Canny Edge Detection

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/

1 of 89

2 of 89

3 of 89

4 of 89

5 of 89

6 of 89

7 of 89

8 of 89

9 of 89

10 of 89

11 of 89

12 of 89

13 of 89

14 of 89

15 of 89

16 of 89

17 of 89

18 of 89

19 of 89

20 of 89

21 of 89

22 of 89

23 of 89

24 of 89

25 of 89

26 of 89

27 of 89

28 of 89

29 of 89

30 of 89

31 of 89

32 of 89

33 of 89

34 of 89

35 of 89

36 of 89

37 of 89

38 of 89

39 of 89

40 of 89

41 of 89

42 of 89

43 of 89

44 of 89

45 of 89

46 of 89

47 of 89

48 of 89

49 of 89

50 of 89

51 of 89

52 of 89

53 of 89

54 of 89

55 of 89

56 of 89

57 of 89

58 of 89

59 of 89

60 of 89

61 of 89

62 of 89

63 of 89

64 of 89

65 of 89

66 of 89

67 of 89

68 of 89

69 of 89

70 of 89

71 of 89

72 of 89

73 of 89

74 of 89

75 of 89

76 of 89

77 of 89

78 of 89

79 of 89

80 of 89