CSE 5524: �Object detection and segmentation

1

Homework and quiz plan

• HW 3: (10%)
• Release: 3/18
• Due: 4/1

​

• HW 4: (10 – 20%)
• Release: 4/1
• Due 1: 4/15
• Due 2: 4/22

​

• Quiz: 4% coming soon

Final project (20 – 30%)

• Team forming:
• 2 – 3 students: same expectation

​

• Tentative plan:
• Team forming: 2/28
• Project sketch: 3/7 (2%) – 1 page at most: what you plan to do, who your teammates are. Figures are OK to be added.
• Project proposal: 3/20 (3%)
• Project presentation: 4/22 & 4/23 (10%)
• Project report & code release: 4/25 (15%)

​

Project first glance

• Pre-defined tasks:
• CVPR 2025 competition: https://cvpr.thecvf.com/Conferences/2025/workshop-list
• Keywords: competition & challenge
• Example: https://sites.google.com/view/fgvc12/competitions

​

• Reproducing existing algorithms
• Benchmarking existing algorithms
• Reproducing examples in the textbook

​

• Self-defined “research” tasks:
• Need approval
• Need justification

​

​

​

​

​

Today

• CNN recap
• Semantic segmentation
• Object detection

​

​

5

Convolutional neural networks (CNN) for detection

6

Convolutional neural networks (CNN)

7

Receptive field

8

Linear receptive field

Exponential receptive field

(with pooling + down-sampling)

Layers of feature maps (representations)

9

What does a large response at each layer/channel mean?

Short summary

A general architecture of CNN or visual transformers involves

• Multiple layers of computations + nonlinearity + (pooling + striding)
• These result in a (final) feature map
• The map goes through FC layers (MLP)
• Usually, we keep the network till the feature map

​

10

Popular CNN architectures

• Encoder, decoder

11

Popular CNN architectures

• Encoder + decoder for segmentation

12

Popular CNN architectures

• U-Net: Encoder + decoder + skip links

13

Today

• CNN recap
• Semantic segmentation
• Object detection

​

14

Exemplar computer vision tasks

[C. Rieke, 2019]

Exemplar computer vision tasks

Retrieval, representation learning

Image generation

Vision and language

Neural Radiance Fields (NeRF)

Object-centric vs. scene-centric images

​

​

​

​

​

​

17

Object-centric images:

• contain a single class of objects
• The object size is usually large
• The background is simple

Scene-centric images:

• contain multiple classes of objects
• The object sizes can vary
• The background is challenging
• Objects may be occluded

ImageNet [image-level label]:

• 1K classes (~1M images)
• 21K classes (~14M images)

MSCOCO [instance segment]:

• 82 classes (~0.3M images)

Classification on object-centric images

​

​

​

​

• Single object class (not multi-label cases)

​

• Properties to capture:
• Translation invariant
• Rotation invariant (left-right flip)
• Scale invariant

car

elephant

Representative 2D recognition tasks

• “Same” input: images

​

• “Different” outputs:
• A C-dim class probability vector
• A set of bounding boxes, each with box location and class probability
• An W x H x C feature map
• A combination of b) and c)

​

• “Different” labeled training data

​

19

W

H

a)

c)

b)

d)

Object- vs. scene centric images

20

MSCOCO [scene-centric]:

• Instance-level label
• 82 classes (~0.3M images)

ImageNet [object-centric]:

• Image-level class label
• 1K classes (~1M images)
• 21K classes (~14M images)

Object- vs. scene centric images

• Object-centric images usually contain a single class of objects.
• Object frequency and semantic cues in different kinds of images can be different!

21

Semantic segmentation

• Every “pixel” to have a class label

​

• Properties:
• High-resolution output
• Context
• Localization
• Multi-scale

22

New architecture?

23

Single spatial output!

Fully-convolutional network (FCN)

24

Feature map

Vector after vectorization

Dog

Cat

Boat

Bird

Matrix multiplication, inner product

Fully-convolutional network (FCN)

25

Dog

Cat

Boat

Bird

Feature map

Fully-convolutional network (FCN)

26

[Long et al., Fully Convolutional Networks for Semantic Segmentation, CVPR 2015]

Up-sampling

27

Interpolation

Deconvolution

Fully-convolutional network (FCN)

28

Help localization

Help

context + semantics

[Long et al., Fully Convolutional Networks for Semantic Segmentation, CVPR 2015]

U-Net

29

Help localization

Help

context + semantics

[Ronneberger et al., U-Net: Convolutional Networks for Biomedical Image Segmentation, MICCAI 2015]

U-Net (aka, Hourglass network)

30

Dilated (Atrous) convolution

Exponential receptive field:�w/o down-sampling + up-sampling

w/ same # of parameters to learn

CRF to improve localization

• DeepLab

32

CRF: similar and nearby pixels have the same class label

[Chen et al., DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, PAMI 2017]

Atrous Spatial Pyramid Pooling (ASPP)�for multi-scale features

33

Example results

34

[Nirkin et al., HyperSeg, 2021]

Ground truth

[Zhao et al., Pyramid scene parsing network, 2017]

Today

• CNN recap
• Semantic segmentation
• Object detection

​

​

35

Object detection

• Properties:

​

• Labels + bounding boxes
• Localization
• Multi-scale
• Context
• “Undetermined” numbers

​

​

36

[class, u-center, v-center, width, height]

Naïve way

• Sliding window
• Time consuming
• What size?

37

R-CNN

• Objectness proposal
• CNN classifier
• Box refinement

38

[Girshick et al., Rich feature hierarchies for accurate object detection and semantic segmentation, CVPR 2014]

Selective search for proposal generation

• Step 1:
• Not deep learning
• super-pixel-based segmentation

​

• Step 2:
• Recursively combine similar regions into larger ones

​

• Step 3:
• Boxes fitting

39

[Stanford CS 231b]

R-CNN

40

[Girshick et al., Rich feature hierarchies for accurate object detection and semantic segmentation, CVPR 2014]

[Girshick, CVPR 2019 tutorial]

R-CNN

• Box regression:
• (du, dv)
• (dw, dh)

​

By offset = MLP(feature)

41

Proposal

Ground truth

R-CNN

• Problems:
• Slow: every proposal needs to go through a “full” CNN
• Mis-detection: the proposal algorithm is not trained together

42

Fast R-CNN

43

ROI pooling

[Girshick, CVPR 2019 tutorial]

[Girshick, Fast R-CNN, ICCV 2015]

ROI pooling vs. ROI align

44

ROI Align

ROI Pooling

Making features extracted from different proposals the same size!

Faster R-CNN

45

ROI pooling

[Girshick, CVPR 2019 tutorial]

[Ren et al., Faster r-cnn: Towards real-time object detection with region proposal networks, NIPS 2015]

How to develop RPN�(region proposal network)?

46

5 * 8 * K * (2 + 4)

[Ren et al., 2015]

Ground truth

Anchor

What do we learn from RPN?

• “How to encode your labeled data so that your CNN can learn from them and predict them” is important!

​

• Inference: predict these “values” and accordingly transfer them to bounding boxes!

​

47

Questions?

How to deal with object sizes?

49

[Lin et al., Feature Pyramid Networks for Object Detection, CVPR 2017]

Mask R-CNN

50

[Girshick, CVPR 2019 tutorial]

[He et al., Mask r-cnn, ICCV 2017]

Mask R-CNN: for instance segmentation

51

CNN: convolutional neural network

RPN: region proposal network

2-stage vs. 1-stage detectors

• Other names: single-shot, single-pass, … (e.g., YOLO, SSD)
• Difference: no ROI pooling/align

52

[Redmon et al., 2016]

2-stage detector

1-stage detector

Exemplar 1-stage detectors

53

[Liu et al., 2016]

SSD

YOLO

[Redmon et al., 2016]

Exemplar 1-stage detectors (Retina Net)

54

[Lin et al., 2017]

2-stage vs. 1-stage detectors

• Pros for 1-stage:
• Faster!

​

• Cons for 1-stage:
• Too many negative locations; scale

55

[Redmon et al., 2016]

Inference: choose few from many

• Non-Maximum Suppression (NMS)

56

[Pictures from “towards data science” post]

Example results

57

[Zhang, et al., 2021]

New approach to object detection

• DEtection Transformer [Nicolas Carion et al.]

New approach to object detection

• DEtection Transformer [Nicolas Carion et al.]

Key names

• Tsung-Yi Lin
• Ross Girshick
• Kaiming He
• Piotr Dollar

Take home

• Good tutorials online:
• CVPR 2017-2022, ECCV 2018-2020, ICCV 2017-2021 [search tutorial or workshop]
• ICML/NeurIPS/ICLR 2018-2021 [search tutorial or workshop]

​

• Good framework:
• PyTorch: Torchvision
• PyTorch: Detectron2

​

• Good source code:
• Papers with code: https://paperswithcode.com/

61

LiDAR-based 3D perception

LiDAR-based 3D perception

63

[Source: Graham Murdoch/Popular Science]

LiDAR:

• Light Detection and Ranging sensor
• accurate 3D point clouds of the environment

LiDAR-based 3D perception

You can view the LiDAR point clouds from different angles

64

Frontal view

Bird’s-eye view (BEV)

Two major ways to process LiDAR point clouds

• Point-wise processing
• PointNet [Qi et al., 2017]
• PointNet++ [Qi et al., 2017]
• PointRCNN [Shi et al., 2019]
• …

​

• Voxel-based processing: turn points into a tensor (e.g., W x D x H x F)
• PointPillars [Lang et al., 2019]
• VoxelNet [Zhou et al., 2017]
• PIXOR [Liang et al., 2018]
• …

65

Voxel-based processing + 3D object detectors

• Occupation (PIXOR): 3D points as a 3D occupation tensor from bird’s-eye-view

66

[Yang et al., PIXOR: Real-time 3D Object Detection from Point Clouds, 2019]

height

depth

Left-right