1 of 92

Lecture 15: Object detection

1

Contains slides from S. Lazebnik, R. Girshick, B. Hariharan

Contains slides from S. Lazebnik, R. Girshick, B. Hariharan, P. Isola

2 of 92

  • Introduction to scene understanding
  • Object detection models
  • Evaluating object detectors
  • Shape understanding

2

Today

3 of 92

  • Introduction to scene understanding
  • Object detection models
  • Evaluating object detectors
  • Shape understanding

3

Today

4 of 92

Before we talk about objects:

What does it mean to understand a scene?

4

5 of 92

5

Image contains Photoshopped sign

Source: Antonio Torralba

6 of 92

6

Source: Antonio Torralba

7 of 92

7

DUCK

DUCK

GRASS

PERSON

TREE

LAKE

BENCH

PERSON

PERSON

PERSON

DUCK

PATH

SKY

SIGN

Source: Antonio Torralba

8 of 92

8

Park

Source: Antonio Torralba

9 of 92

9

A view of a park on a nice spring day

Source: Antonio Torralba

10 of 92

10

DUCKS ON TOP

OF THE GRASS

PEOPLE UNDER THE

SHADOW OF THE TREES

Source: Torralba, Freeman, Isola

11 of 92

11

PERSON FEEDING�DUCKS IN THE PARK

PEOPLE WALKING IN THE PARK

DUCKS LOOKING FOR FOOD

Do not � feed �the ducks

sign

Source: Antonio Torralba

12 of 92

What makes this challenging?

12

13 of 92

Why do we care about recognition?

We can perceive the 3D shape, texture, material properties, without knowing about objects. But, the concept of category encapsulates also information about what can we do with those objects.

13

“We therefore include the perception of function as a proper –indeed, crucial- subject for vision science”, from Vision Science, chapter 9, Palmer.

Source: Torralba, Freeman, Isola

14 of 92

14

Object categories aren’t everything

sky

building

flag

wall

banner

bus

cars

bus

face

street lamp

Slide by Fei Fei, Fergus & Torralba

15 of 92

15

sky

building

flag

wall

banner

bus

cars

bus

face

street lamp

A picture is worth a 1000 words…

Or just 10?

Source: A. Efros

Object categories aren’t everything

16 of 92

What labels? Recognizing exact instances?

16

A Beijing City Transit Bus #17, serial number 43253?

Source: A. Efros

17 of 92

Need more general (useful) information

Functional:

    • A large vehicle that may be moving fast, probably to the right, and will hurt you if you stand in its way.
    • However, at specified places, it will allow you to enter it and transport you quickly over large distances.

Communicational:

    • bus, autobus, λεωφορείο, ônibus, автобус, 公共汽车, etc.

17

What can we say the very

first time we see this thing?

Source: A. Efros

18 of 92

Visual challenges with categories

  • A lot of categories are functional

​

​

  • Categories are 3D, but images are 2D

​

  • World is highly varied

Chair

car

train

Source: A. Efros

19 of 92

Limits to direct perception

19

20 of 92

Importance of context

20

Source: Antonio Torralba

21 of 92

  • Introduction to scene understanding
  • Object detection models
  • Evaluating object detectors
  • Shape understanding

21

Today

22 of 92

22

Previously: object recognition

“Birds”

Source: Torralba, Freeman, Isola

23 of 92

23

Previously: semantic segmentation

“A bunch of bird stuff”

Source: Torralba, Freeman, Isola

24 of 92

24

Object detection

Bird

Bird

Bird

Challenge: unbounded number of detections, possibly multiple detections per pixel

Classification and localization

Each bounding box is:�[x,y,w,h]

Source: Torralba, Freeman, Isola

25 of 92

Idea #1: regress bounding box

25

Contains slides from S. Lazebnik, R. Girshick, B. Hariharan

Recall: template matching

dog

26 of 92

27 of 92

Idea #2: sliding window

27

Contains slides from S. Lazebnik, R. Girshick, B. Hariharan

Recall: template matching

1

​

dog

duck

no object

0

 

Bounding box

Need multiple scales and aspect ratios

28 of 92

Idea #2: sliding window

28

Contains slides from S. Lazebnik, R. Girshick, B. Hariharan

Recall: template matching

1

​

dog

duck

no object

0

 

Bounding box

29 of 92

Example: histograms of oriented gradients (HOG)

29

Image credit: N. Snavely

Source: S. Lazebnik

N. Dalal and B. Triggs, Histograms of Oriented Gradients for Human Detection, CVPR 2005

30 of 92

Example: pedestrian detection with HOG

Train a pedestrian template using a linear classifier. Represent each window using HOG.

30

N. Dalal and B. Triggs, Histograms of Oriented Gradients for Human Detection, CVPR 2005

positive training examples

negative training examples

Source: S. Lazebnik

31 of 92

Pedestrian detection with HOG

​

For multi-scale detection, repeat over multiple levels of a HOG pyramid

31

Template

HOG feature map

Detector response map

Source: S. Lazebnik

32 of 92

Idea #3: selective search

 

32

Contains slides from S. Lazebnik, R. Girshick, B. Hariharan

Recall: template matching

Source: S. Lazebnik

33 of 92

Selective search

  • Example: edge boxes [Zitnick & Dollar, 2014]�
  • Heuristic: detect edges, group them into contours

​

  • Rank each window based on number of contours in window

​

  • These are the only windows our detector will see

33

Contains slides from S. Lazebnik, R. Girshick, B. Hariharan

Recall: template matching

34 of 92

Recall: idea #3: selective search

 

34

Contains slides from S. Lazebnik, R. Girshick, B. Hariharan

Recall: template matching

Source: S. Lazebnik

35 of 92

Selective search

  • Example: edge boxes [Zitnick & Dollar, 2014]�
  • Heuristic: detect edges, group them into contours

​

  • Rank each window based on number of contours in window

​

  • These are the only windows our detector will see

35

Contains slides from S. Lazebnik, R. Girshick, B. Hariharan

Recall: template matching

36 of 92

R-CNN: Region proposals + CNN features

36

Input image

ConvNet

ConvNet

ConvNet

Linear

Linear

Linear

Warped image regions

Forward each region through a CNN

Classify regions with linear classifier

Region proposals from selective search (~2K rectangles that are likely to contain objects)

R. Girshick, J. Donahue, T. Darrell, and J. Malik, Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, CVPR 2014.

Source: R. Girshick

37 of 92

R-CNN at test time

37

Input

image

Extract region

proposals (~2k / image)

Compute CNN

features

a. Crop

Source: R. Girshick

38 of 92

R-CNN at test time

38

Input

image

Extract region

proposals (~2k / image)

Compute CNN

features

a. Crop

b. Scale

227 x 227

Source: R. Girshick

39 of 92

R-CNN at test time

39

1. Crop

b. Scale

Input

image

Extract region

proposals (~2k / image)

Compute CNN

features

c. Forward propagate

Output: “fc7” features

Source: R. Girshick

40 of 92

R-CNN at test time

40

Input

image

Extract region

proposals (~2k / image)

Compute CNN

features

Warped proposal

4096-dimensional

fc7 feature vector

linear classifier

person? 1.6

horse? -0.3

...

...

Classify

regions

Source: R. Girshick

41 of 92

Proposal refinement

41

Linear regression

​

on CNN features

Original

proposal

Predicted

object bounding box

Bounding-box regression

Source: R. Girshick

42 of 92

Bounding-box regression

42

original

predicted

Δh × h + h

Δw × w + w

(Δx × w + x,

Δy × h + h)

w

h

(x, y)

Source: R. Girshick

43 of 92

Non-maximum suppression

0.9

0.8

Source: B. Hariharan

If two boxes overlap significantly (e.g. > 50% IoU), drop the one with the lower score. Usually use greedy algorithm.

44 of 92

Problems with R-CNN

  1. Slow! Have to run CNN per window
  2. Hand-crafted mechanism for region proposal might be suboptimal.

44

ConvNet

ConvNet

ConvNet

Linear

Linear

Linear

45 of 92

“Fast” R-CNN: reuse features between proposals

45

ConvNet

Forward whole image through ConvNet

Conv5 feature map of image

RoI Pooling layer

Linear +

softmax

FCs

Fully-connected layers

Softmax classifier

Region proposals

Linear

Bounding-box regressors

R. Girshick, Fast R-CNN, ICCV 2015

Source: R. Girshick

46 of 92

ROI Pooling

  • How do we crop from a feature map?
  • Step 1: Resize boxes to account for subsampling

46

Layer 1

Layer 2

Layer 3

Source: B. Hariharan

47 of 92

ROI Pooling

  • How do we crop from a feature map?
  • Step 2: Snap to feature map grid

47

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Source: B. Hariharan

48 of 92

ROI Pooling

  • How do we crop from a feature map?
  • Step 3: Overlay a new grid of fixed size

48

​

​

​

​

​

​

​

​

​

Source: B. Hariharan

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

49 of 92

ROI Pooling

  • How do we crop from a feature map?
  • Step 4: Take max in each cell
  • Can improve with bilinear sampling

49

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Source: B. Hariharan

Classification

50 of 92

“Faster” R-CNN: learn region proposals

50

CNN

feature map

Region proposals

CNN

feature map

Region Proposal Network

share features

51 of 92

RPN: Region Proposal Network

51

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

 

Conv feature map

Source: R. Girshick

52 of 92

RPN: Region Proposal Network

52

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

3x3 “sliding window”

Scans the feature map

looking for objects

 

Conv feature map

Source: R. Girshick

53 of 92

RPN: Anchor Box

53

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Anchor box: predictions are �w.r.t. this box, not the 3x3�sliding window

Conv feature map

 

3x3 “sliding window”

Scans the feature map

looking for objects

Source: R. Girshick

54 of 92

RPN: Anchor Box

54

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

3x3 “sliding window”

  • Objectness classifier [0, 1]

​

  • Box regressor� predicting (dx, dy, dh, dw)

Conv feature map

 

Anchor box: predictions are �w.r.t. this box, not the 3x3�sliding window

Source: R. Girshick

55 of 92

RPN: Prediction (on object)

55

3x3 “sliding window”

  • Objectness classifier [0, 1]

​

  • Box regressor� predicting (dx, dy, dh, dw)

P(object) = 0.94

Objectness score

Source: R. Girshick

56 of 92

RPN: Prediction (on object)

56

Anchor box: transformed by

box regressor

3x3 “sliding window”

  • Objectness classifier [0, 1]

​

  • Box regressor� predicting (dx, dy, dh, dw)

P(object) = 0.94

Source: R. Girshick

57 of 92

RPN: Prediction (off object)

57

Anchor box: transformed by

box regressor

3x3 “sliding window”

  • Objectness classifier

​

  • Box regressor� predicting (dx, dy, dh, dw)

P(object) = 0.02

Objectness score

Source: R. Girshick

58 of 92

RPN: Multiple Anchors

58

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

3x3 “sliding window”

  • K objectness classifiers

​

  • K box regressors

Conv feature map

 

Anchor boxes: K anchors�per location with different�scales and aspect ratios

Source: R. Girshick

59 of 92

One network, four losses

59

image

CNN

feature map

Region Proposal Network

proposals

RoI pooling

Classification loss

Bounding-box regression loss

…

Classification loss

Bounding-box regression loss

Source: R. Girshick, K. He, S. Lazebnik

60 of 92

Faster R-CNN results

60

Source: S. Lazebnik

61 of 92

How do we deal with scale?

Idea #1: Gaussian pyramid

61

Feature pyramid

Gaussian image pyramid

[Lin et al., “Feature Pyramid Networks for Object Detection”, 2017]

Source: Torralba, Freeman, Isola

62 of 92

Image and features pyramids

62

Each pooling reduces the resolution by

a factor of 2

VGG network

CNN architectures build:

  • Multiscale feature hierarchies, but
  • each layer builds a different representation
  • first layers are low level, while
  • last layers are high level.

A feature pyramid requires a uniform representations across scales.

Source: Torralba, Freeman, Isola

63 of 92

Idea #2: Feature pyramid network

63

Image pyramid

Encoder-decoder architecture (U-Net)

Feature pyramid

Source: Torralba, Freeman, Isola

64 of 92

Object detection progress

64

R-CNNv1

Fast R-CNN

Before CNNs

After CNNs

Faster R-CNN

Performance on PASCAL VOC

Source: S. Lazebnik

65 of 92

Streamlined detection architectures

  • The Faster R-CNN pipeline separates proposal generation and region classification:

65

Conv feature map of the entire image

Region Proposals

RoI features

RPN

RoI pooling

Classification + Regression

Detections

Source: S. Lazebnik

Conv feature map of the entire image

Detections

Classification + Regression

  • Is it possible do detection in one shot?

66 of 92

Single-stage object detector

  • Divide the image into a coarse grid using a fully convolutional net
  • Directly predict class label, confidence, and a few candidate boxes for each grid cell.

66

Source: S. Lazebnik

J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, You Only Look Once: Unified, Real-Time Object Detection, CVPR 2016

67 of 92

YOLO detector

  1. Take convolutional feature maps at 7x7 resolution
  2. Predict, at each location, a score for each class and 2 bounding boxes (w/ confidence)
    • E.g. for 20 classes, output is 7x7x30 (30 = 20 + 2*(4+1))

67

Source: S. Lazebnik

  • 7x speedup over Faster R-CNN (45-155 FPS vs. 7-18 FPS) but less accurate (e.g. 65% vs. 72 mAP%)
  • Extension: use anchor boxes in last layer to try a few possible aspect ratios

J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, You Only Look Once: Unified, Real-Time Object Detection, CVPR 2016

68 of 92

69 of 92

70 of 92

Object Detection: Grounding DINO 2023

Trained on Small Numbers of Text Labels

Detects Objects Based on Any Text Descriptions

71 of 92

  • Introduction to scene understanding
  • Object detection models
  • Evaluating object detectors
  • Shape understanding

71

Today

72 of 92

Evaluating an object detector

  •  

72

cat

dog

cat: 0.8

dog: 0.6

dog: 0.55

Ground truth (GT)

Source: S. Lazebnik

73 of 92

Evaluating an object detector

73

Source: B. Hariharan

Intersection over union (also known as Jaccard similarity)

74 of 92

Evaluating an object detector

  • For each class, plot Precision-Recall curve and compute Average Precision (area under the curve)
  • Take mean of AP over classes to get mAP

74

Precision:

true positive detections / �total detections

Recall:

true positive detections / �total positive test instances

Source: S. Lazebnik

75 of 92

Average precision

75

true positive detections / �total positive test instances

Precision

1

Recall

Source: B. Hariharan

true pos. / �total detections

76 of 92

Average precision

76

true positive detections / �total positive test instances

Precision

1

1

Recall

Source: B. Hariharan

true pos. / �total detections

77 of 92

  • Introduction to scene understanding
  • Object detection models
  • Evaluating object detectors
  • Shape understanding

​

77

Today

78 of 92

Beyond bounding boxes: instance segmentation

78

Predict segmentation mask for each object

From COCO [Lin et al., 2014]

Source: B. Hariharan

79 of 92

Instance segmentation

79

[He et al., “Mask R-CNN”, 2017]

ROI pooling with tiny change:

bilinear interpolation instead of max

Faster R-CNN

Extra “head” on network predicts binary mask

80 of 92

Example Mask Training Targets

80

Image with training proposal

28x28 mask target

Image with training proposal

28x28 mask target

Source: R. Girshick

81 of 92

Example Mask Training Targets

81

Image with training proposal

28x28 mask target

Image with training proposal

28x28 mask target

Source: R. Girshick

82 of 92

Example Mask Training Targets

82

Image with training proposal

28x28 mask target

Image with training proposal

28x28 mask target

Source: R. Girshick

83 of 92

Example Mask Training Targets

83

Image with training proposal

28x28 mask target

Image with training proposal

28x28 mask target

84 of 92

85 of 92

86 of 92

Human Pose

  • Add keypoint head (28x28x17)

​

  • Predict one “mask” for each keypoint

​

  • Softmax over spatial locations (encodes one keypoint per mask “prior”)

keypoints

x17

(Not shown: Head architecture is slightly different for keypoints)

17 keypoint “mask”

predictions shown as�heatmaps with OKS�scores from argmax�positions

Source: R. Girshick

87 of 92

88 of 92

89 of 92

Next time: language

89

90 of 92

We still need lots of labeled examples

90

Mask R-CNN on COCO with Different Training Set Sizes

Image source: R. Girshick

91 of 92

Handle the long tail of the distribution

91

Object categories

Frequency

Person, dog, table, …

Teacup, wreath, birdfeeder, …

92 of 92

Handle the “long tail” of the distribution

92

From COCO (80 categories)

[Lin et al., 2014]

LVIS dataset (1000+ categories)

“Few shot” (e.g. < 20 examples)

[Gupta et al., 2019]

Image source: R. Girshick