1
Hidden
Outputs
Input
LSTM
LSTM
CNN
LSTM
LSTM
LSTM
LSTM
LSTM
“A”
“clown”
“swimming”
“fish”
“in”
“open”
LSTM
“seas”
“A”
“clown”
“fish”
“swimming”
“in”
“open”
“seas”
END
Targets
Maximum likelihood: maximize probability of all ground truth words.
Source: Isola, Torralba, Freeman
2
LSTM
CNN
Hidden
Outputs
Input
Samples
LSTM
“clown”
LSTM
“A”
LSTM
“fish”
LSTM
“swimming”
LSTM
“in”
LSTM
“open”
LSTM
“seas”
“A”
“clown”
“fish”
“swimming”
“in”
“open”
“seas”
END
Sample from predicted distribution over words.
Alternatively, sample most likely word.
Source: Isola, Torralba, Freeman
3
Captioning: popular topic circa 2015
Source: D. Parikh
Show and Tell: A Neural Image Caption Generator
4
[Vinyals et. al., CVPR 2015]
Source: Isola, Torralba, Freeman
Show and Tell: A Neural Image Caption Generator
5
[Vinyals et. al., CVPR 2015]
6
Transformer-based captioning
7
[Desai and Johnson, VirTex, 2020]
Good source of features
8
[Desai and Johnson, VirTex, 2020]
9
2016
10
[http://www.visualqa.org/challenge.html]
11
12
Questions and answers collected with Amazon Mechanical Turk
Architecture
13
Image
Question
Answer
often, we work with word embeddings, rather than one-hot representations of words
Source: Isola, Torralba, Freeman
Architecture
14
There are 1000 possible answers in this system. Questions are unlimited.
[Agrawal et al., "VQA: Visual Question Answering" 2016]
15
16
17
18
19
20
What’s going on?
21
The Giraffe-Tree problem
22
[“Measuring Machine Intelligence Through Visual Question Answering”, Zitnick et al., 2016]
Nearest neighbor baseline
23
Test
Train
A small white stuffed bear on a blanket.�A stuffed bear that is sitting in a bed.�A white teddy bear with a blue tie sitting on a couch.
Pink and orange teddy bears are lined up.
A little girl and her mom with a frosted cookie.
A young girl holding a stuffed animal near other dolls.
A teddy bear sitting on the floor.
.
.
.
Source: L. Zitnick
24
A black and white cat sitting in a bathroom sink.
Two zebras and a giraffe in a field.
Nearest Neighbor
See mscoco.org for image information
Source: L. Zitnick
Image captioning
25
A man riding a
motorcycle on a beach.
An airplane is parked on the
tarmac at an airport.
Source: L. Zitnick
Results
26
| CIDEr-D | Meteor | ROUGE-L | BLEU-4 |
Google[4] | 0.943 | 0.254 | 0.53 | 0.309 |
MSR Captivator[9] | 0.931 | 0.248 | 0.526 | 0.308 |
m-RNN[15] | 0.917 | 0.242 | 0.521 | 0.299 |
MSR[8] | 0.912 | 0.247 | 0.519 | 0.291 |
Nearest Neighbor[11] | 0.886 | 0.237 | 0.507 | 0.280 |
m-RNN (Baidu/ UCLA)[16] | 0.886 | 0.238 | 0.524 | 0.302 |
Berkeley LRCN[2] | 0.869 | 0.242 | 0.517 | 0.277 |
Human[5] | 0.854 | 0.252 | 0.484 | 0.217 |
Montreal/Toronto[10] | 0.85 | 0.243 | 0.513 | 0.268 |
PicSOM[13] | 0.833 | 0.231 | 0.505 | 0.281 |
MLBL[7] | 0.74 | 0.219 | 0.499 | 0.26 |
ACVT[1] | 0.709 | 0.213 | 0.483 | 0.246 |
NeuralTalk[12] | 0.674 | 0.21 | 0.475 | 0.224 |
Tsinghua Bigeye[14] | 0.673 | 0.207 | 0.49 | 0.241 |
MIL[6] | 0.666 | 0.214 | 0.468 | 0.216 |
Brno University[3] | 0.517 | 0.195 | 0.403 | 0.134 |
COCO Caption Challenge
Source: L. Zitnick
Visual Question Answering Dataset
Source: L. Zitnick
28
Aside: biases in data collection
29
Source: Isola, Torralba, Freeman
Getting more humans in the annotation loop
30
Labeling to get a Ph.D.
Labeling for money�(Sorokin, Forsyth, 2008)
Labeling because it �gives you added value
Visipedia�(Belongie, Perona, et al)
Labeling for fun
Luis Von Ahn and Laura Dabbish 2004
Labeling to prove you’re human
Source: Isola, Torralba, Freeman
Beware of the human in your loop
31
Let’s check a few simple experiments
Source: Isola, Torralba, Freeman
People have biases…
32
Turkers were offered 1 cent to pick a number from 1 to 10.
From http://groups.csail.mit.edu/uid/deneme/
~850 turkers
Experiment by Greg Little
Source: Isola, Torralba, Freeman
Do humans have consistent biases?
33
From http://groups.csail.mit.edu/uid/deneme/
Experiment by Greg Little
Source: Isola, Torralba, Freeman
Are humans reliable even in simple tasks?
34
From http://groups.csail.mit.edu/uid/deneme/
Experiment by Greg Little
A: 2
B: 96
C: 2
Results of 100 HITS:
Source: Isola, Torralba, Freeman
Do humans do what you ask for?
35
31 heads, 19 tails
After 50 HITS:
34 heads, 16 tails
And 50 more:
From http://groups.csail.mit.edu/uid/deneme/
Experiment by Rob Miller
Source: Isola, Torralba, Freeman
36
But suppose we messed up. Our test setting doesn’t look like the training data!
So we can sometimes collect good training data.
How can we bridge the domain gap?
Source: Isola, Torralba, Freeman
Finding more representative images
[Fouhey et al., "From Lifestyle Vlogs to Everyday Actions", 2017]
Finding more representative images
[Fouhey et al., "From Lifestyle Vlogs to Everyday Actions", 2017]
Name that dataset game
39
[Torralba and Efros, “An unbiased look at dataset bias,” 2011]
Modern architectures
Learning representations from language
[Radford et al., ”CLIP” , 2021]
maximize:
Contrastive learning
“Zero-shot” classification
[Radford et al., ”CLIP” , 2021]
(1) Create classifier from label text
(2) Test how well each prompt fits an image
“Zero-shot” classification
[Radford et al., ”CLIP” , 2021]
“Zero-shot” classification
[Radford et al., ”CLIP” , 2021]
“Zero-shot” classification
[Radford et al., ”CLIP” , 2021]
Image-to-text
“A giraffe standing in the grass next to a tree”
Text-to-image
“A giraffe standing in the grass next to a tree”
Text-to-image
[Ramesh et al., “Zero-Shot Text-to-Image Generation”, 2021]
Text-to-image
[Ramesh et al., “Zero-Shot Text-to-Image Generation”, 2021]
Diffusion text-to-image synthesis
a teddy bear on a skateboard in times square
A photo of Michelangelo’s sculpture of David wearing headphones djing
“A sea otter with a pearl earring” by Johannes Vermeer
3D render of a cute tropical fish in an aquarium on a dark blue background, digital art
[“DALL-E 2”, Ramesh et al., 2022]
Diffusion text-to-image synthesis
[Ramesh et al., DALL-E 2, 2021]
1. Train CLIP
Diffusion text-to-image synthesis
[Ramesh et al., DALL-E 2, 2021]
2. Estimate image embedding from text embedding
Diffusion text-to-image synthesis
[Ramesh et al., DALL-E 2, 2021]
3. Conditional model
Conditional diffusion
add noise
add noise
Some extra details: we also scale the image by a factor
5. Language is a powerful form of supervision
6. Language is a natural “user interface” for computer vision systems
Summary
Adapted from Isola, Freeman, Torralba
Next class: GANs
56