1 of 62

CSE 5524: �Image processing – 2

2 of 62

Today

  • Recap
  • Image processing (chapter 15)
  • Fourier analysis (chapter 16)

​

​

​

​

2

3 of 62

Image is a discrete signal

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

4 of 62

System is to process signal

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

5 of 62

Linear system

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

6 of 62

Linear system (mathematical definition)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

7 of 62

Linear system (in neural network)

  • Fully connected layer

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

8 of 62

Linear translation invariant (LTI) system

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

9 of 62

Convolution is an LTI system

  • h[n] is named a convolution kernel
  • Input-output relationship: linear weighted sum; weights depend on relative positions

​

​

​

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

10 of 62

Relationship?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

11 of 62

Relationship?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

12 of 62

Relationship

Linear layer

Convolutional layer

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

13 of 62

Revisit convolutions in the context of CNNs

0

0

0

0

1

0

0

0

1

1

0

0

1

1

1

0

1

1

1

1

1

1

1

1

1

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

0

0

1

0

1

1

1

1

1

Feature map (nodes) at layer t

Feature map at layer t+1

“Filter” weights

(3-by-3)

Inner product

Element-wise multiplication and sum

1

14 of 62

Revisit convolutions in the context of CNNs

14

0

0

0

0

1

0

0

0

1

1

0

0

1

1

1

0

1

1

1

1

1

1

1

1

1

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

0

0

1

0

1

1

1

1

1

“Filter” weights

(3-by-3)

Inner product

6

Feature map (nodes) at layer t

Feature map at layer t+1

15 of 62

2D case

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

16 of 62

Today

  • Recap
  • Image processing (chapter 15)
  • Fourier analysis (chapter 16)

​

​

​

​

16

17 of 62

Properties of convolutions

  • Communicative

​

  • Associative

​

  • Distributed

​

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

18 of 62

Properties of convolutions

  • Identity

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

19 of 62

Some examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

20 of 62

Some examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

21 of 62

Some examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

22 of 62

Some examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

23 of 62

Some examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

24 of 62

Some examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

25 of 62

Boundaries

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

26 of 62

Cross-correlation vs. convolution

  • Cross correlation:

​

  • Convolution:

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

27 of 62

Cross-correlation vs. convolution

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Template Matching

28 of 62

Today

  • Recap
  • Image processing (chapter 15)
  • Fourier analysis (chapter 16)

​

​

​

​

28

29 of 62

Music and frequency

30 of 62

Music and frequency

Fourier transform

31 of 62

Fourier transform

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

32 of 62

Fourier transform

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

33 of 62

Fourier transform

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

34 of 62

Fourier transform

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

35 of 62

Fourier transform

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

36 of 62

Fourier transform

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

37 of 62

Fourier transform

  •  

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

38 of 62

Fourier transform

  •  

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

39 of 62

Continuous and discrete waves

  • Continuous sine:
  • Discrete sine:

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

40 of 62

Discrete waves

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

k = 1

k = 2

k = 3

41 of 62

Sine and cosine in 2D

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

42 of 62

Sine and cosine in 2D

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

43 of 62

** Complex exponential **

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

44 of 62

** Complex exponential **

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

45 of 62

Revisiting Fourier transform

  •  

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

46 of 62

Discrete Fourier transform (DFT) and inverse (IDFT)

Input image

Periodic functions (bases)

Pixel position

Frequency

Frequency response

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

47 of 62

Discrete Fourier transform (DFT) and inverse (IDFT)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

48 of 62

Visualization

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

DFT

Image

Image

49 of 62

Visualization – images at different frequencies

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

50 of 62

Visualization – different frequency combinations

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

51 of 62

Compression

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

52 of 62

Visualization – disregard some frequencies

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

53 of 62

Convolution and Fourier

  • Convolution in time/space:
  • Applying Fourier transform

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

54 of 62

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

55 of 62

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

56 of 62

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

57 of 62

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

58 of 62

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

59 of 62

Modulation

  • Convolution in time/space equals multiplication in frequencies
  • Multiplication in frequencies equals convolution in frequencies

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

60 of 62

Fun editing by mixing

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

61 of 62

Amplitude vs. Phase

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

62 of 62

Image processing by filters

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]