Lecture 15: Object detection
1
Contains slides from S. Lazebnik, R. Girshick, B. Hariharan
Contains slides from S. Lazebnik, R. Girshick, B. Hariharan, P. Isola
2
Today
3
Today
Before we talk about objects:
What does it mean to understand a scene?
4
5
Image contains Photoshopped sign
Source: Antonio Torralba
6
Source: Antonio Torralba
7
DUCK
DUCK
GRASS
PERSON
TREE
LAKE
BENCH
PERSON
PERSON
PERSON
DUCK
PATH
SKY
SIGN
Source: Antonio Torralba
8
Park
Source: Antonio Torralba
9
A view of a park on a nice spring day
Source: Antonio Torralba
10
DUCKS ON TOP
OF THE GRASS
PEOPLE UNDER THE
SHADOW OF THE TREES
Source: Torralba, Freeman, Isola
11
PERSON FEEDING�DUCKS IN THE PARK
PEOPLE WALKING IN THE PARK
DUCKS LOOKING FOR FOOD
Do not � feed �the ducks
sign
Source: Antonio Torralba
What makes this challenging?
12
Why do we care about recognition?
We can perceive the 3D shape, texture, material properties, without knowing about objects. But, the concept of category encapsulates also information about what can we do with those objects.
13
“We therefore include the perception of function as a proper –indeed, crucial- subject for vision science”, from Vision Science, chapter 9, Palmer.
Source: Torralba, Freeman, Isola
14
Object categories aren’t everything
sky
building
flag
wall
banner
bus
cars
bus
face
street lamp
Slide by Fei Fei, Fergus & Torralba
15
sky
building
flag
wall
banner
bus
cars
bus
face
street lamp
A picture is worth a 1000 words…
Or just 10?
Source: A. Efros
Object categories aren’t everything
What labels? Recognizing exact instances?
16
A Beijing City Transit Bus #17, serial number 43253?
Source: A. Efros
Need more general (useful) information
Functional:
Communicational:
17
What can we say the very
first time we see this thing?
Source: A. Efros
Visual challenges with categories
Chair
car
train
Source: A. Efros
Limits to direct perception
19
Importance of context
20
Source: Antonio Torralba
21
Today
22
Previously: object recognition
“Birds”
Source: Torralba, Freeman, Isola
23
Previously: semantic segmentation
“A bunch of bird stuff”
Source: Torralba, Freeman, Isola
24
Object detection
Bird
Bird
Bird
Challenge: unbounded number of detections, possibly multiple detections per pixel
Classification and localization
Each bounding box is:�[x,y,w,h]
Source: Torralba, Freeman, Isola
Idea #1: regress bounding box
25
Contains slides from S. Lazebnik, R. Girshick, B. Hariharan
Recall: template matching
dog
Idea #2: sliding window
27
Contains slides from S. Lazebnik, R. Girshick, B. Hariharan
Recall: template matching
1
dog
duck
no object
0
Bounding box
Need multiple scales and aspect ratios
Idea #2: sliding window
28
Contains slides from S. Lazebnik, R. Girshick, B. Hariharan
Recall: template matching
1
dog
duck
no object
0
Bounding box
Example: histograms of oriented gradients (HOG)
29
Image credit: N. Snavely
Source: S. Lazebnik
N. Dalal and B. Triggs, Histograms of Oriented Gradients for Human Detection, CVPR 2005
Example: pedestrian detection with HOG
Train a pedestrian template using a linear classifier. Represent each window using HOG.
30
N. Dalal and B. Triggs, Histograms of Oriented Gradients for Human Detection, CVPR 2005
positive training examples
negative training examples
Source: S. Lazebnik
Pedestrian detection with HOG
For multi-scale detection, repeat over multiple levels of a HOG pyramid
31
Template
HOG feature map
Detector response map
Source: S. Lazebnik
Idea #3: selective search
32
Contains slides from S. Lazebnik, R. Girshick, B. Hariharan
Recall: template matching
Source: S. Lazebnik
Selective search
33
Contains slides from S. Lazebnik, R. Girshick, B. Hariharan
Recall: template matching
Recall: idea #3: selective search
34
Contains slides from S. Lazebnik, R. Girshick, B. Hariharan
Recall: template matching
Source: S. Lazebnik
Selective search
35
Contains slides from S. Lazebnik, R. Girshick, B. Hariharan
Recall: template matching
R-CNN: Region proposals + CNN features
36
Input image
ConvNet
ConvNet
ConvNet
Linear
Linear
Linear
Warped image regions
Forward each region through a CNN
Classify regions with linear classifier
Region proposals from selective search (~2K rectangles that are likely to contain objects)
R. Girshick, J. Donahue, T. Darrell, and J. Malik, Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, CVPR 2014.
Source: R. Girshick
R-CNN at test time
37
Input
image
Extract region
proposals (~2k / image)
Compute CNN
features
a. Crop
Source: R. Girshick
R-CNN at test time
38
Input
image
Extract region
proposals (~2k / image)
Compute CNN
features
a. Crop
b. Scale
227 x 227
Source: R. Girshick
R-CNN at test time
39
1. Crop
b. Scale
Input
image
Extract region
proposals (~2k / image)
Compute CNN
features
c. Forward propagate
Output: “fc7” features
Source: R. Girshick
R-CNN at test time
40
Input
image
Extract region
proposals (~2k / image)
Compute CNN
features
Warped proposal
4096-dimensional
fc7 feature vector
linear classifier
person? 1.6
horse? -0.3
...
...
Classify
regions
Source: R. Girshick
Proposal refinement
41
Linear regression
on CNN features
Original
proposal
Predicted
object bounding box
Bounding-box regression
Source: R. Girshick
Bounding-box regression
42
original
predicted
Δh × h + h
Δw × w + w
(Δx × w + x,
Δy × h + h)
w
h
(x, y)
Source: R. Girshick
Non-maximum suppression
0.9
0.8
Source: B. Hariharan
If two boxes overlap significantly (e.g. > 50% IoU), drop the one with the lower score. Usually use greedy algorithm.
Problems with R-CNN
44
ConvNet
ConvNet
ConvNet
Linear
Linear
Linear
“Fast” R-CNN: reuse features between proposals
45
ConvNet
Forward whole image through ConvNet
Conv5 feature map of image
RoI Pooling layer
Linear +
softmax
FCs
Fully-connected layers
Softmax classifier
Region proposals
Linear
Bounding-box regressors
R. Girshick, Fast R-CNN, ICCV 2015
Source: R. Girshick
ROI Pooling
46
Layer 1
Layer 2
Layer 3
Source: B. Hariharan
ROI Pooling
47
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
Source: B. Hariharan
ROI Pooling
48
| | |
| | |
| | |
Source: B. Hariharan
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
ROI Pooling
49
| | |
| | |
| | |
| | |
| | |
| | |
Source: B. Hariharan
See more here: https://deepsense.ai/region-of-interest-pooling-explained/
Classification
“Faster” R-CNN: learn region proposals
50
CNN
feature map
Region proposals
CNN
feature map
Region Proposal Network
S. Ren, K. He, R. Girshick, and J. Sun, Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, NIPS 2015
share features
RPN: Region Proposal Network
51
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
Conv feature map
Source: R. Girshick
RPN: Region Proposal Network
52
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
3x3 “sliding window”
Scans the feature map
looking for objects
Conv feature map
Source: R. Girshick
RPN: Anchor Box
53
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
Anchor box: predictions are �w.r.t. this box, not the 3x3�sliding window
Conv feature map
3x3 “sliding window”
Scans the feature map
looking for objects
Source: R. Girshick
RPN: Anchor Box
54
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
3x3 “sliding window”
Conv feature map
Anchor box: predictions are �w.r.t. this box, not the 3x3�sliding window
Source: R. Girshick
RPN: Prediction (on object)
55
3x3 “sliding window”
P(object) = 0.94
Objectness score
Source: R. Girshick
RPN: Prediction (on object)
56
Anchor box: transformed by
box regressor
3x3 “sliding window”
P(object) = 0.94
Source: R. Girshick
RPN: Prediction (off object)
57
Anchor box: transformed by
box regressor
3x3 “sliding window”
P(object) = 0.02
Objectness score
Source: R. Girshick
RPN: Multiple Anchors
58
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
3x3 “sliding window”
Conv feature map
Anchor boxes: K anchors�per location with different�scales and aspect ratios
Source: R. Girshick
One network, four losses
59
image
CNN
feature map
Region Proposal Network
proposals
RoI pooling
Classification loss
Bounding-box regression loss
…
Classification loss
Bounding-box regression loss
Source: R. Girshick, K. He, S. Lazebnik
Faster R-CNN results
60
Source: S. Lazebnik
How do we deal with scale?
Idea #1: Gaussian pyramid
61
Feature pyramid
Gaussian image pyramid
[Lin et al., “Feature Pyramid Networks for Object Detection”, 2017]
Source: Torralba, Freeman, Isola
Image and features pyramids
62
Each pooling reduces the resolution by
a factor of 2
VGG network
CNN architectures build:
A feature pyramid requires a uniform representations across scales.
Source: Torralba, Freeman, Isola
Idea #2: Feature pyramid network
63
Image pyramid
Encoder-decoder architecture (U-Net)
Feature pyramid
Source: Torralba, Freeman, Isola
Object detection progress
64
R-CNNv1
Fast R-CNN
Before CNNs
After CNNs
Faster R-CNN
Performance on PASCAL VOC
Source: S. Lazebnik
Streamlined detection architectures
65
Conv feature map of the entire image
Region Proposals
RoI features
RPN
RoI pooling
Classification + Regression
Detections
Source: S. Lazebnik
Conv feature map of the entire image
Detections
Classification + Regression
Single-stage object detector
66
Source: S. Lazebnik
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, You Only Look Once: Unified, Real-Time Object Detection, CVPR 2016
YOLO detector
67
Source: S. Lazebnik
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, You Only Look Once: Unified, Real-Time Object Detection, CVPR 2016
Object Detection: Grounding DINO 2023
Trained on Small Numbers of Text Labels
Detects Objects Based on Any Text Descriptions
71
Today
Evaluating an object detector
72
cat
dog
cat: 0.8
dog: 0.6
dog: 0.55
Ground truth (GT)
Source: S. Lazebnik
Evaluating an object detector
73
Source: B. Hariharan
Intersection over union (also known as Jaccard similarity)
Evaluating an object detector
74
Precision:
true positive detections / �total detections
Recall:
true positive detections / �total positive test instances
Source: S. Lazebnik
Average precision
75
true positive detections / �total positive test instances
Precision
1
Recall
Source: B. Hariharan
true pos. / �total detections
Average precision
76
true positive detections / �total positive test instances
Precision
1
1
Recall
Source: B. Hariharan
true pos. / �total detections
77
Today
Beyond bounding boxes: instance segmentation
78
Predict segmentation mask for each object
From COCO [Lin et al., 2014]
Source: B. Hariharan
Instance segmentation
79
[He et al., “Mask R-CNN”, 2017]
ROI pooling with tiny change:
bilinear interpolation instead of max
Faster R-CNN
Extra “head” on network predicts binary mask
Example Mask Training Targets
80
Image with training proposal
28x28 mask target
Image with training proposal
28x28 mask target
Source: R. Girshick
Example Mask Training Targets
81
Image with training proposal
28x28 mask target
Image with training proposal
28x28 mask target
Source: R. Girshick
Example Mask Training Targets
82
Image with training proposal
28x28 mask target
Image with training proposal
28x28 mask target
Source: R. Girshick
Example Mask Training Targets
83
Image with training proposal
28x28 mask target
Image with training proposal
28x28 mask target
Human Pose
keypoints
x17
(Not shown: Head architecture is slightly different for keypoints)
17 keypoint “mask”
predictions shown as�heatmaps with OKS�scores from argmax�positions
Source: R. Girshick
Next time: language
89
We still need lots of labeled examples
90
Mask R-CNN on COCO with Different Training Set Sizes
Image source: R. Girshick
Handle the long tail of the distribution
91
Object categories
Frequency
Person, dog, table, …
Teacup, wreath, birdfeeder, …
Handle the “long tail” of the distribution
92
From COCO (80 categories)
[Lin et al., 2014]
LVIS dataset (1000+ categories)
“Few shot” (e.g. < 20 examples)
[Gupta et al., 2019]
Image source: R. Girshick