1 of 91

1

EC 500 A1: �Camera Modeling (Cont.)

& Image Processing

2 of 91

2

Self introduction

3 of 91

Today

​

  • Recap
  • Camera model & calibration: cont.
  • Image processing

3

4 of 91

What is the data structure of images?

  • Ordered array (matrix) of pixel values:

​

​

​

​

  • A set of pixels and their locations:
    • Make geometry explicit!

4

Regular spatial grid

 

5 of 91

Homogeneous and heterogeneous coordinates

5

heterogeneous

homogeneous

6 of 91

Chaining transformations, using homogeneous system

  • Homogeneous coordinates allow us to combine transformations via products

6

7 of 91

From 3D to 2D and back: two 3D coordinate systems

7

Image

Image formation

Know how to convert coordinates

Know how to convert coordinates back

8 of 91

Homogeneous coordinates for 3D points

  • Homogeneous and heterogeneous coordinates

8

heterogeneous

homogeneous

9 of 91

Perspective projection in homogeneous coordinates

9

 

 

10 of 91

Camera-intrinsic parameters

10

(X, Y, Z)

n

m

11 of 91

11

Questions?

12 of 91

Camera-extrinsic parameters

12

R: 3D rotation between coordinate systems

​

T: 3D translation between coordinate systems

 

13 of 91

Camera-extrinsic parameters

13

heterogeneous

homogeneous

14 of 91

Camera-extrinsic parameters

14

homogeneous

 

 

 

 

15 of 91

15

Questions?

16 of 91

Full camera model (intrinsic + extrinsic)

16

 

17 of 91

Full camera model (intrinsic + extrinsic)

Camera

3D

Intrinsic

Rotation

Translation

18 of 91

18

Questions?

19 of 91

Glance at camera calibration

  •  

19

20 of 91

Going from 2D to 3D

  • If we know Z of (x, y), can we recover (X, Y, Z)?

20

21 of 91

What is 3D reconstruction nowadays?

https://depth-anything-v2.github.io/

22 of 91

What is 3D reconstruction nowadays?

https://vgg-t.github.io/

23 of 91

[Image credit: Sooyoung Jeon]

24 of 91

[Image credit: Sooyoung Jeon]

25 of 91

Today

​

  • Recap
  • Camera model & calibration: cont.
  • Image processing

25

26 of 91

Today

  • Signals and images (Chapter 15)
  • Linear systems
  • Linear translation invariant (LTI) systems and convolutions
  • Frequency response and Fourier transform (Chapter 16)

​

​

​

​

26

27 of 91

Notations (self-check)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

28 of 91

Signal and system

  • A signal is a measurement of some physical quantity
  • A system is a process/function that transforms a signal into another

28

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Neural network image classifier

29 of 91

Continue and discrete signal

29

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Sampling

Sampling period

30 of 91

Image?

31 of 91

Image is a discrete signal

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

…

…

…

…

…

 

32 of 91

Why do we study images as signals?

32

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

33 of 91

Why do we study images as signals?

33

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

34 of 91

Geometric transformation vs. image processing

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

35 of 91

Today

  • Recap
  • Signals and images (Chapter 15)
  • Linear systems
  • Linear translation invariant (LTI) systems and convolutions
  • Frequency response and Fourier transform (Chapter 16)

​

​

​

​

35

36 of 91

System is to process signal

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

37 of 91

Linear system

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

38 of 91

Linear system (mathematical definition)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

39 of 91

You saw this before

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

40 of 91

Linear system: 1D & 2D

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

2D signal (e.g., image)

1D signal (e.g., sound)

n

k

[n, m]

k

l

41 of 91

Today

  • Recap
  • Signals and images (Chapter 15)
  • Linear systems
  • Linear translation invariant (LTI) systems and convolutions
  • Frequency response and Fourier transform (Chapter 16)

​

​

​

​

41

42 of 91

Linear translation invariant (LTI) system

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

43 of 91

Linear translation invariant (LTI) system

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

 

44 of 91

Linear translation invariant (LTI) system

Bird detection

[140, 25; 30, 20]

Bird detection

[40, 100; 30, 20]

Bird detection

45 of 91

Convolution is an LTI system

  • h[n] is named a convolution kernel (or filter)
  • Input-output relationship: linear weighted sum; weights depend on relative positions
  • Different h process the image differently

​

​

​

45

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

46 of 91

Relationship?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Linear

LTI

47 of 91

Relationship?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Linear

LTI

48 of 91

Relationship?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Linear

LTI

49 of 91

Relationship

Linear

Convolutional

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

50 of 91

50

Questions?

51 of 91

LTI system: 1D & 2D

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

1D signal (e.g., sound)

n

k

n

 

52 of 91

LTI system: 1D & 2D

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

2D signal (e.g., image)

53 of 91

2D convolutions

53

Convolution kernel

(3-by-3)

1

1

1

1

1

0

1

0

0

0

0

0

0

1

0

0

0

1

1

0

0

1

1

1

0

1

1

1

1

1

1

1

1

1

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Convolutions =

Flip + Inner product

6

Input image

Output image

0

0

1

0

1

1

1

1

1

54 of 91

2D convolutions

0

0

0

0

1

0

0

0

1

1

0

0

1

1

1

0

1

1

1

1

1

1

1

1

1

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

1

0

0

1

0

1

1

1

1

1

Convolutions =

Flip + Inner product

Input image

Output image

55 of 91

2D convolutions

55

Convolution kernel

(3-by-3)

1

1

1

0

0

0

1

1

1

0

0

0

0

1

0

0

0

1

1

0

0

1

1

1

0

1

1

1

1

1

1

1

1

1

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Convolutions =

Flip + Inner product

4

Input image

Output image

56 of 91

Convolution example

56

1

1

1

0

0

0

1

1

1

57 of 91

57

Questions?

58 of 91

The convolution computation

  •  

n

k

n

 

 

59 of 91

The convolution computation

  •  

 

 

n=0

 

 

n=0

 

 

 

 

 

 

 

60 of 91

The convolution computation

  •  

 

n=0

n=0

n=0

n=6

61 of 91

61

Questions?

62 of 91

Convolution for translation

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

63 of 91

Some examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

64 of 91

Some examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

65 of 91

Some examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

66 of 91

Some examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

67 of 91

Some examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

68 of 91

Some examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

69 of 91

69

Questions?

70 of 91

Convolution

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Template Matching

71 of 91

Which one can be achieved by convolutions?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

72 of 91

Today

  • Signals and images (Chapter 15)
  • Linear systems
  • Linear translation invariant (LTI) systems and convolutions
  • Frequency response and Fourier transform (Chapter 16)
    • Basic
    • Illustration
    • Convolution and modulation
    • Math

​

​

​

​

72

73 of 91

Music and frequency

74 of 91

Music and frequency

Fourier transform

75 of 91

Fourier transform

75

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

76 of 91

Fourier transform

76

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

77 of 91

Fourier transform

77

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

A signal can be represented by a linear combination of “periodic” functions!

78 of 91

78

Questions?

79 of 91

Frequency in images

80 of 91

Frequency in images

81 of 91

Goal

Spatial

DFT: frequency

Amplitude: A

 

82 of 91

Today

  • Signals and images (Chapter 15)
  • Linear systems
  • Linear translation invariant (LTI) systems and convolutions
  • Frequency response and Fourier transform (Chapter 16)
    • Basic
    • Illustration
    • Convolution and modulation
    • Math

​

​

​

​

82

83 of 91

Fourier transform

83

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

A signal can be represented by a linear combination of “periodic” functions!

84 of 91

Fourier transform

  •  

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

85 of 91

Fourier transform

  •  

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

“inner products”

“periodic” functions!

86 of 91

Continuous and discrete waves

  • Continuous sine:
  • Discrete sine:

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

87 of 91

Discrete waves

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

k = 1

k = 2

k = 3

88 of 91

88

Questions?

89 of 91

Sine and cosine in 2D

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

90 of 91

Sine and cosine in 2D

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

91 of 91

Frequency response: amplitude and phases

91

m

n

m

 

 

Fourier-like

transform

Amplitude

Phase

u

u

v

v

u

u

v

v

n