1 of 74

CSE 5524: �Image processing - 1

2 of 74

Today

  • Neural networks (chapter 12 & 13)
  • Image processing (chapter 15)

​

​

​

​

2

3 of 74

Naïve neural networks for image classification

3

[Gif credits: Gradient descent 3Blue1Brown series S3 E2]

4 of 74

Perceptron

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

 

 

 

 

 

 

5 of 74

Perceptron

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

 

 

 

 

 

 

6 of 74

Perceptron

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

 

 

 

 

 

 

7 of 74

Perceptron

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

 

 

 

 

 

 

8 of 74

Multi-layer perceptron (MLP)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Element-wise

9 of 74

Activations vs. Parameters

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

10 of 74

Fast activation and slow parameters

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

11 of 74

Fast activation and slow parameters

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

12 of 74

Why do we need activation?

  •  

13 of 74

Questions?

14 of 74

Deep Nets

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

15 of 74

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

16 of 74

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

17 of 74

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

18 of 74

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

19 of 74

Questions?

20 of 74

Deep nets are universal approximators

  • When it is deep or wider enough

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

depth

width

True function

Step-wise approximation

21 of 74

Deep nets are universal approximators

  • When it is deep or wider enough

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

depth

width

True function

Step-wise approximation

22 of 74

Deep nets are universal approximators

  • When it is deep or wider enough

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

depth

width

True function

Step-wise approximation

23 of 74

Deep learning: learning with neural nets

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

What are neural nets? What is deep learning?

24 of 74

Deep learning: learning with neural nets

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

25 of 74

Questions?

26 of 74

Data structure (in training)

  • Image classification problem

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

27 of 74

Data structure (in training)

  • Image classification problem

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

28 of 74

Data structure (in training)

  • Image classification problem

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

29 of 74

Data structure (in training)

  • Image classification problem

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

30 of 74

Data structure (in training)

  • Image classification problem

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Stochastic Gradient Descent

31 of 74

Data structure (in training)

  • Image classification problem

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Stochastic Gradient Descent

32 of 74

Data structure (in training)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Batch size = 3

33 of 74

Data structure (in training)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Batch size = 3

34 of 74

Questions?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

35 of 74

Layers

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

36 of 74

Layers

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

37 of 74

Reading

  • Read 12.7.3
  • Read 12.7.4
  • Read 12.8

38 of 74

Why are neural networks powerful?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

39 of 74

Geometric “transformation” perspective

Function

Data transformation

Data mapping

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

40 of 74

What is this function?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

41 of 74

What is this function?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

42 of 74

What is this function?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

43 of 74

What is this function?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

44 of 74

Put together

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

45 of 74

2D examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

 

Observation?

46 of 74

2D examples

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

 

Which one is more “diverse”?

47 of 74

What does a neural network classifier want to do?

  •  

48 of 74

Visualizing a neural network classifier (binary)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

 

49 of 74

Visualizing a neural network classifier (binary)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

50 of 74

How is a neural network trained?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

51 of 74

Why are neural networks powerful?

52 of 74

Today

  • Neural networks (chapter 12 & 13)
  • Image processing (chapter 15)

​

​

​

​

52

53 of 74

Notations

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

54 of 74

Signal and image

  • A signal is a measurement of some physical quantity
  • A system is a process/function that transforms a signal into another

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

55 of 74

Continue and discrete signal

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Sampling

Sampling period

56 of 74

Image?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

57 of 74

Why do we study images as signals?

58 of 74

Why do we study images as signals?

0

0

0

0

1

0

0

0

1

1

0

0

1

1

1

0

1

1

1

1

1

1

1

1

1

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

0

0

1

0

1

1

1

1

1

Feature map (nodes) at layer t

Feature map at layer t+1

“Filter” weights

(3-by-3)

Inner product

Element-wise multiplication and sum

1

59 of 74

Properties of signals

  • Mean (DC) value

​

​

  • Energy

​

​

  • Difference (squared Euclidean distance)

​

​

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

60 of 74

System

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

61 of 74

Linear system

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

62 of 74

Linear system (mathematical definition)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

63 of 74

Linear system (in neural network)

  • Fully connected layer

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

64 of 74

Linear system for images

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

65 of 74

Linear system for images

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

66 of 74

Linear translation invariant (LTI) system

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

67 of 74

Linear translation invariant (LTI) system

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

What is it?

68 of 74

Convolution

  • h[n] is named a convolution kernel
  • Input-output relationship: linear weighted sum; weights depend on relative positions

​

​

​

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

69 of 74

Relationship?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

70 of 74

Relationship?

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

71 of 74

Relationship

Linear layer

Convolutional layer

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

72 of 74

Recap

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

73 of 74

2D case

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

74 of 74

2D cases

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]