1 of 56

1

Hidden

Outputs

Input

LSTM

LSTM

CNN

LSTM

LSTM

LSTM

LSTM

LSTM

“A”

“clown”

“swimming”

“fish”

“in”

“open”

LSTM

“seas”

“A”

“clown”

“fish”

“swimming”

“in”

“open”

“seas”

END

Targets

Maximum likelihood: maximize probability of all ground truth words.

Source: Isola, Torralba, Freeman

2 of 56

2

LSTM

CNN

Hidden

Outputs

Input

Samples

LSTM

“clown”

LSTM

“A”

LSTM

“fish”

LSTM

“swimming”

LSTM

“in”

LSTM

“open”

LSTM

“seas”

“A”

“clown”

“fish”

“swimming”

“in”

“open”

“seas”

END

Sample from predicted distribution over words.

Alternatively, sample most likely word.

Source: Isola, Torralba, Freeman

3 of 56

3

Captioning: popular topic circa 2015

Source: D. Parikh

4 of 56

Show and Tell: A Neural Image Caption Generator

4

[Vinyals et. al., CVPR 2015]

Source: Isola, Torralba, Freeman

5 of 56

Show and Tell: A Neural Image Caption Generator

5

[Vinyals et. al., CVPR 2015]

6 of 56

6

7 of 56

Transformer-based captioning

7

[Desai and Johnson, VirTex, 2020]

8 of 56

Good source of features

8

[Desai and Johnson, VirTex, 2020]

9 of 56

9

2016

10 of 56

10

[http://www.visualqa.org/challenge.html]

11 of 56

11

12 of 56

12

Questions and answers collected with Amazon Mechanical Turk

13 of 56

Architecture

13

Image

Question

Answer

often, we work with word embeddings, rather than one-hot representations of words

Source: Isola, Torralba, Freeman

14 of 56

Architecture

14

There are 1000 possible answers in this system. Questions are unlimited.

[Agrawal et al., "VQA: Visual Question Answering" 2016]

15 of 56

15

16 of 56

16

17 of 56

17

18 of 56

18

19 of 56

19

20 of 56

20

21 of 56

What’s going on?

21

22 of 56

The Giraffe-Tree problem

22

[“Measuring Machine Intelligence Through Visual Question Answering”, Zitnick et al., 2016]

23 of 56

Nearest neighbor baseline

23

Test

Train

A small white stuffed bear on a blanket.�A stuffed bear that is sitting in a bed.�A white teddy bear with a blue tie sitting on a couch.

Pink and orange teddy bears are lined up.

A little girl and her mom with a frosted cookie.

A young girl holding a stuffed animal near other dolls.

A teddy bear sitting on the floor.

.

.

.

Source: L. Zitnick

24 of 56

24

A black and white cat sitting in a bathroom sink.

Two zebras and a giraffe in a field.

Nearest Neighbor

See mscoco.org for image information

Source: L. Zitnick

25 of 56

Image captioning

25

A man riding a

motorcycle on a beach.

An airplane is parked on the

tarmac at an airport.

Source: L. Zitnick

26 of 56

Results

26

​

CIDEr-D

Meteor

ROUGE-L

BLEU-4

Google[4]

0.943

0.254

0.53

0.309

MSR Captivator[9]

0.931

0.248

0.526

0.308

m-RNN[15]

0.917

0.242

0.521

0.299

MSR[8]

0.912

0.247

0.519

0.291

Nearest Neighbor[11]

0.886

0.237

0.507

0.280

m-RNN (Baidu/ UCLA)[16]

0.886

0.238

0.524

0.302

Berkeley LRCN[2]

0.869

0.242

0.517

0.277

Human[5]

0.854

0.252

0.484

0.217

Montreal/Toronto[10]

0.85

0.243

0.513

0.268

PicSOM[13]

0.833

0.231

0.505

0.281

MLBL[7]

0.74

0.219

0.499

0.26

ACVT[1]

0.709

0.213

0.483

0.246

NeuralTalk[12]

0.674

0.21

0.475

0.224

Tsinghua Bigeye[14]

0.673

0.207

0.49

0.241

MIL[6]

0.666

0.214

0.468

0.216

Brno University[3]

0.517

0.195

0.403

0.134

COCO Caption Challenge

Source: L. Zitnick

27 of 56

Visual Question Answering Dataset

Source: L. Zitnick

28 of 56

28

Aside: biases in data collection

29 of 56

29

Source: Isola, Torralba, Freeman

30 of 56

Getting more humans in the annotation loop

30

Labeling to get a Ph.D.

Labeling for money�(Sorokin, Forsyth, 2008)

Labeling because it �gives you added value

Visipedia�(Belongie, Perona, et al)

Labeling for fun

Luis Von Ahn and Laura Dabbish 2004

Labeling to prove you’re human

Source: Isola, Torralba, Freeman

31 of 56

Beware of the human in your loop

  • What do you know about them?
  • Will they do the work you pay for?

31

Let’s check a few simple experiments

Source: Isola, Torralba, Freeman

32 of 56

People have biases…

32

Turkers were offered 1 cent to pick a number from 1 to 10.

From http://groups.csail.mit.edu/uid/deneme/

~850 turkers

Experiment by Greg Little

Source: Isola, Torralba, Freeman

33 of 56

Do humans have consistent biases?

33

From http://groups.csail.mit.edu/uid/deneme/

Experiment by Greg Little

Source: Isola, Torralba, Freeman

34 of 56

Are humans reliable even in simple tasks?

34

From http://groups.csail.mit.edu/uid/deneme/

Experiment by Greg Little

A: 2

B: 96

C: 2

Results of 100 HITS:

Source: Isola, Torralba, Freeman

35 of 56

Do humans do what you ask for?

35

31 heads, 19 tails

After 50 HITS:

34 heads, 16 tails

And 50 more:

From http://groups.csail.mit.edu/uid/deneme/

Experiment by Rob Miller

Source: Isola, Torralba, Freeman

36 of 56

36

But suppose we messed up. Our test setting doesn’t look like the training data!

So we can sometimes collect good training data.

How can we bridge the domain gap?

Source: Isola, Torralba, Freeman

37 of 56

Finding more representative images

[Fouhey et al., "From Lifestyle Vlogs to Everyday Actions", 2017]

38 of 56

Finding more representative images

[Fouhey et al., "From Lifestyle Vlogs to Everyday Actions", 2017]

39 of 56

Name that dataset game

39

[Torralba and Efros, “An unbiased look at dataset bias,” 2011]

40 of 56

Modern architectures

41 of 56

Learning representations from language

[Radford et al., ”CLIP” , 2021]

 

maximize:

Contrastive learning

42 of 56

“Zero-shot” classification

[Radford et al., ”CLIP” , 2021]

(1) Create classifier from label text

(2) Test how well each prompt fits an image

43 of 56

“Zero-shot” classification

[Radford et al., ”CLIP” , 2021]

44 of 56

“Zero-shot” classification

[Radford et al., ”CLIP” , 2021]

45 of 56

“Zero-shot” classification

[Radford et al., ”CLIP” , 2021]

46 of 56

Image-to-text

“A giraffe standing in the grass next to a tree”

47 of 56

Text-to-image

“A giraffe standing in the grass next to a tree”

48 of 56

Text-to-image

[Ramesh et al., “Zero-Shot Text-to-Image Generation”, 2021]

49 of 56

Text-to-image

[Ramesh et al., “Zero-Shot Text-to-Image Generation”, 2021]

50 of 56

Diffusion text-to-image synthesis

a teddy bear on a skateboard in times square

A photo of Michelangelo’s sculpture of David wearing headphones djing

“A sea otter with a pearl earring” by Johannes Vermeer

3D render of a cute tropical fish in an aquarium on a dark blue background, digital art

[“DALL-E 2”, Ramesh et al., 2022]

51 of 56

Diffusion text-to-image synthesis

[Ramesh et al., DALL-E 2, 2021]

1. Train CLIP

52 of 56

Diffusion text-to-image synthesis

[Ramesh et al., DALL-E 2, 2021]

2. Estimate image embedding from text embedding

53 of 56

Diffusion text-to-image synthesis

[Ramesh et al., DALL-E 2, 2021]

3. Conditional model

54 of 56

Conditional diffusion

add noise

add noise

 

 

Some extra details: we also scale the image by a factor

 

55 of 56

  1. Deep nets learn representations
  2. This is useful because representations transfer — they act as prior knowledge that enables quick learning on new tasks
  3. Representations can also be learned without labels
  4. Without labels there are many ways to learn representations. We saw:
    1. representations as compressed codes
    2. representations that are predictive of their context

5. Language is a powerful form of supervision

6. Language is a natural “user interface” for computer vision systems

Summary

Adapted from Isola, Freeman, Torralba

56 of 56

Next class: GANs

56