1 of 40

CSE 163

Convolutions�

Suh Young Choi�

🎶 Listening to: Shang-Chi soundtrack

💬 Before Class: Buffalo buffalo buffalo buffalo Buffalo buffalo buffalo. Weird how that’s a grammatically correct sentence.

2 of 40

Announcements

Reading Assignment 5 now out!

“On the Dangers of Stochastic Parrots” by Bender et al. (2021)

THA 5 due on Thursday (last one of the quarter!)

Remaining Project/Portfolio Components

Part 3: Report/Code (50pts, due March 9)
Part 4: Presentation (20pts, done in section on March 12)
Part 5: Peer Feedback (20pts, due March 15)

10pts from Section Assignment 10 🡨 Go to section!!!
10pts from Individual Assignment in Gradescope

2

3 of 40

Bonus Resubmission Opportunity

Possibility of Bonus Resubmission Period

BRP would be from 3/11 – 3/13
Submit any ONE Technical Component + any ONE Creative Component
Doesn’t have to be for the same THA
Course evaluation response rate must be >65% by 3/9 to get the BRP
Course evals go out on 3/2 (Ed announcement will be made)

3

4 of 40

Remaining Due Dates

To assist with planning:

4

Sun	Mon	Tue	Wed	Thu	Fri	Sat
Feb 22	23	24	25	26 THA 5 due	27	28
Mar 1	2 RA 5 due	3 Resubs due (THA 4)	4	5	6	7
8	9 CP 5 due Report/Code due	10 Resubs due (THA 5)	11	12 Presentations (in section)	13 Last day to submit CP/RA	14
15 Peer feedback due	16	17	18	19 Final Reflection	20	21

5 of 40

This Time

Convolutions

Last Time

Numpy arrays
Broadcasting
Images

5

6 of 40

Images as

Matrices

6

Grey-scale images can be represented as matrices.

Grey-scale: 255

Grey-scale: 0

data = imageio.imread(‘...’)�

data[rows, columns] = #

7 of 40

Color Images

When you overlap each color channel, it creates a picture we are used to seeing.

Pixels on your monitor let out specified R/G/B light

7

data[rows, columns, channels] = #

8 of 40

Convolution

When wanting to use “local” information, we commonly use a sliding window approach (i.e. a convolution)

Move the sliding window across the image, and compute the sum of the element wise product of the window (kernel) and image

8

3	3	2	1	0
0	0	1	3	1
3	1	2	2	3
2	0	0	2	2
2	0	0	0	1

Image

0	1	2
2	2	0
0	1	2

Kernel

9 of 40

Convolution Example

9

10 of 40

Convolutions!

an unconvoluted example

11 of 40

9	8	7
6	5	4
3	2	1

Image

2	1

Kernel

12 of 40

9	8	7
6	5	4
3	2	1

Image

2	1

Output

Kernel

13 of 40

9	8	7
6	5	4
3	2	1

Image

2	1

Output

2

1

Kernel

14 of 40

18

8

2

1

18	8	7
6	5	4
3	2	1

Image

2	1

Output

Kernel

15 of 40

8

18

2

1

18	8	7
6	5	4
3	2	1

Image

2	1

Output

26

Kernel

16 of 40

8

18

2

1

18	8	7
6	5	4
3	2	1

Image

2	1

Output

26

Kernel

17 of 40

8

18

2

1

9	8	7
6	5	4
3	2	1

Image

2	1

Output

26

2

1

Kernel

18 of 40

2

1

8

18

2

1

9	16	7
6	5	4
3	2	1

Image

2	1

Output

16

7

26

Kernel

19 of 40

7

16

2

1

8

18

2

1

9	16	7
6	5	4
3	2	1

Image

2	1

Output

26

23

Kernel

20 of 40

7

16

2

1

8

18

2

1

9	16	7
6	5	4
3	2	1

Image

2	1

Output

26

23

Kernel

21 of 40

7

16

2

1

8

18

2

1

9	8	7
6	5	4
3	2	1

Image

2	1

Output

26

23

2

1

17

Kernel

22 of 40

7

16

2

1

8

18

2

1

9	8	7
6	5	4
3	2	1

Image

2	1

Output

26

23

2

1

17

Kernel

23 of 40

7

16

2

1

8

18

2

1

9	8	7
6	5	4
3	2	1

Image

2	1

Output

26

23

2

1

17

14

Kernel

24 of 40

7

16

2

1

8

18

2

1

9	8	7
6	5	4
3	2	1

Image

2	1

Output

26

23

2

1

17

14

Kernel

25 of 40

7

16

2

1

8

18

2

1

9	8	7
6	5	4
3	2	1

Image

2	1

Output

26

23

2

1

17

14

8

Kernel

26 of 40

7

16

2

1

8

18

2

1

9	8	7
6	5	4
3	2	1

Image

2	1

Output

26

23

2

1

17

14

8

Kernel

27 of 40

7

16

2

1

8

18

2

1

9	8	7
6	5	4
3	2	1

Image

2	1

Output

26

23

2

1

17

14

8

5

Kernel

28 of 40

7

16

2

1

8

18

2

1

9	8	7
6	5	4
3	2	1

Image

2	1

Output

26

23

2

1

17

14

8

5

Kernel

29 of 40

How many times can we fit the kernel into the image?

“

30 of 40

How many times can we fit the kernel into the image?

“

31 of 40

How many times can we fit the kernel into the image?

2

“

32 of 40

How many times can we fit the kernel into the image?

2

“

33 of 40

How many times can we fit the kernel into the image?

2

3

“

34 of 40

How many times can we fit the kernel into the image?

2

3 x

“

35 of 40

General formula:

image

(3x3)

kernel

(1x2)

output

(3x2)

(image_height – k_height + 1, image_width – k_width + 1)

“

36 of 40

More Generic Formula for Result Size

36

Image

Kernel

width

height

k_height

k_width

(0, 0)

k[0]

k[1]

(1, 0)

…

(width - k_width, 0)

k[width - k_width]

Total of (width - k_width + 1) kernels because of zero based indexing!

So the shape of the result will be (height - k_height + 1, width - k_width + 1)

37 of 40

Common Kernels

What do the numbers in the kernel do?

37

Identity

Edge Detection

Sharpen

Box Blur

38 of 40

Image Classification

For a really long time, image classification was done by painstakingly crafting these features (like edge detectors), by hand.

This kind of worked, but we quickly hit our peak using this method.

Then came the buzz-word… deep learning

38

39 of 40

Image Classification

Is this a solved problem?

We get pretty decent error rates on challenges like ImageNet

What we can’t do

Sometimes can’t generalize to other real-world datasets
Adversarial attacks

39

40 of 40

Before Next Time

Complete Lesson 20

Remember not for points, but do go towards Weekly Tokens

THA 5 due tomorrow
Go to section!

Next Time

Machine learning with images

40