CSE 5539: �Computer Vision

Representative 2D recognition tasks

• “Same” input: images

​

• “Different” outputs:
• A C-dim class probability vector
• A set of bounding boxes, each with box location and class probability
• An W x H x C feature map
• A combination of b) and c)

​

• “Different” labeled training data

​

2

W

H

a)

c)

b)

d)

Semantic segmentation

• Every “pixel” to have a class label

​

• Properties:
• High-resolution output
• Context
• Localization
• Multi-scale

3

U-Net

4

Help localization

Help

context + semantics

[Ronneberger et al., U-Net: Convolutional Networks for Biomedical Image Segmentation, MICCAI 2015]

Today

• Applications: 2D recognition
• 2D object detection
• 2D generation

​

• Practical problems:
• Insufficient (labeled) data
• Domain shifts

​

​

​

Object detection

• Properties:

​

• Labels + bounding boxes
• Localization
• Multi-scale
• Context
• “Undetermined” numbers

​

​

6

[class, u-center, v-center, width, height]

Naïve way

• Sliding window
• Time consuming
• What size?

7

R-CNN

• Objectness proposal
• CNN classifier
• Box refinement

8

[Girshick et al., Rich feature hierarchies for accurate object detection and semantic segmentation, CVPR 2014]

Selective search for proposal generation

• Step 1:
• Not deep learning
• super-pixel-based segmentation

​

• Step 2:
• Recursively combine similar regions into larger ones

​

• Step 3:
• Boxes fitting

9

[Stanford CS 231b]

R-CNN

10

[Girshick et al., Rich feature hierarchies for accurate object detection and semantic segmentation, CVPR 2014]

[Girshick, CVPR 2019 tutorial]

R-CNN

• Box regression:
• (du, dv)
• (dw, dh)

​

By offset = MLP(feature)

11

Proposal

Ground truth

R-CNN

• Problems:
• Slow: every proposal needs to go through a “full” CNN
• Mis-detection: the proposal algorithm is not trained together

12

Fast R-CNN

13

ROI pooling

[Girshick, CVPR 2019 tutorial]

[Girshick, Fast R-CNN, ICCV 2015]

ROI pooling vs. ROI align

14

ROI Align

ROI Pooling

Making features extracted from different proposals the same size!

Faster R-CNN

15

ROI pooling

[Girshick, CVPR 2019 tutorial]

[Ren et al., Faster r-cnn: Towards real-time object detection with region proposal networks, NIPS 2015]

How to develop RPN�(region proposal network)?

16

5 * 8 * K * (2 + 4)

[Ren et al., 2015]

Ground truth

Anchor

What do we learn from RPN?

• “How to encode your labeled data so that your CNN can learn from them and predict them” is important!

​

• Inference: predict these “values” and accordingly transfer them to bounding boxes!

​

17

Questions?

How to deal with object sizes?

19

[Lin et al., Feature Pyramid Networks for Object Detection, CVPR 2017]

Mask R-CNN

20

[Girshick, CVPR 2019 tutorial]

[He et al., Mask r-cnn, ICCV 2017]

Mask R-CNN: for instance segmentation

21

CNN: convolutional neural network

RPN: region proposal network

2-stage vs. 1-stage detectors

• Other names: single-shot, single-pass, … (e.g., YOLO, SSD)
• Difference: no ROI pooling/align

22

[Redmon et al., 2016]

2-stage detector

1-stage detector

Exemplar 1-stage detectors

23

[Liu et al., 2016]

SSD

YOLO

[Redmon et al., 2016]

Exemplar 1-stage detectors (Retina Net)

24

[Lin et al., 2017]

2-stage vs. 1-stage detectors

• Pros for 1-stage:
• Faster!

​

• Cons for 1-stage:
• Too many negative locations; scale

25

[Redmon et al., 2016]

Inference: choose few from many

• Non-Maximum Suppression (NMS)

26

[Pictures from “towards data science” post]

Example results

27

[Zhang, et al., 2021]

New approach to object detection

• DEtection Transformer [Nicolas Carion et al.]

New approach to object detection

• DEtection Transformer [Nicolas Carion et al.]

Key names

• Tsung-Yi Lin
• Ross Girshick
• Kaiming He
• Piotr Dollar

Take home

• Good tutorials online:
• CVPR 2017-2022, ECCV 2018-2020, ICCV 2017-2021 [search tutorial or workshop]
• ICML/NeurIPS/ICLR 2018-2021 [search tutorial or workshop]

​

• Good framework:
• PyTorch: Torchvision
• PyTorch: Detectron2

​

• Good source code:
• Papers with code: https://paperswithcode.com/

31

Today

• Applications: 2D recognition
• 2D object detection
• 2D generation

​

• Practical problems:
• Insufficient (labeled) data
• Domain shifts

​

​

​

Generative models

33

Generative models

Easily samplable distribution

[Credits: Tutorial on Diffusion Models]

What and how to learn?

35

Generative adversarial net (GAN)

36

Generator

Discriminator

[Credits: Mengdi Fan and Xinyu Zhou, CSE 5539 course presentation]

Example results (by Style-GAN)

37

[A Style-Based Generator Architecture for Generative Adversarial Networks, CVPR 2019]

Other generative models

• Denoising Diffusion Probabilistic Models (PPDM)
• Learn to inverse the diffusion process
• Can generate very high-quality images

38

[Denoising Diffusion Probabilistic Models, NeurIPS 2020]

Other generative models

• Denoising Diffusion Probabilistic Models (PPDM)

​

39

[Denoising Diffusion Probabilistic Models, NeurIPS 2020]

Other generative models

40

Big-GAN

Diffusion models

Real

[Diffusion Models Beat GANs on Image Synthesis, NeurIPS 2021]

Conditional image generation

41

[Zhu et al., 2017]

[Wang et al., 2018]

Conditional image generation

42

[Hierarchical Text-Conditional Image Generation with CLIP Latents, arXiv 2022]

Today

• Applications: 2D recognition
• 2D object detection
• 2D generation

​

• Practical problems:
• Insufficient (labeled) data
• Domain shifts

​

​

​

“Some” challenges in DL for CV

• Deep neural networks are “labeled” data hungry
• Mismatch between training and test data

​

44

Recent success in computer vision

[He et al, 2016]

[He et al, 2017]

Challenge-1: Insufficient (labeled) data

• Example: ImageNet-1K (ILSVRC)
• 1,000 object classes
• 1,000 training images per class

​

46

“Top 5” error

It’s hard to collect labeled data

• Collecting and annotating data is time-consuming and expensive

​

• Crowdsourcing can be noisy and may not be feasible for certain problems
• e.g., medical images and applications

​

• For some applications, even “unlabeled” data can be hard to collect
• e.g., fine-grained classes, long-tailed problems, data privacy and protection

​

47

Fine-grained classes

[Credits: Rogerio Feris, ICCV-2019 slides]

Long-tailed distribution

49

Objects in SUN datasets

Collecting dense labels is even harder

• Images with detailed instance segmentation labels

​

​

​

​

​

​

​

• MSCOCO: ~100 classes from 328K images
• Complex tasks, however, have fewer labeled data …

​

​

50

[He et al, 2017]

Collecting dense labels is even harder

• ImageNet: >20K classes and >13M images
• MS-COCO: >0.3M images with captions, instance segments
• Visual Genome: >0.1M images with captions, QA pairs, boxes, etc.

​

​

​

​

​

​

• Complex tasks, however, have fewer labeled data …

​

​

​

[Krishna et al., 2016]

Long-tailed distribution on densely-labeled data

52

Visual Genome

LVIS

Long-tailed distribution

[Liu et al., 2019]

Recap: object- vs. scene centric images

​

​

​

​

​

​

​

• Object frequency can be different!

Long-tailed distribution

Today

• Applications: 2D recognition
• 2D object detection
• 2D generation

​

• Practical problems:
• Insufficient (labeled) data
• Domain shifts

​

​

​

Challenges-2: Domain Shifts

57

KITTI

(Germany)

Argoverse

(USA)

nuScenes

(USA, Singapore)

Lyft

(USA)

Waymo

(USA)

Mismatch between training/test data

• Supervised learning
• (Unknown) distribution: of (x, y): P(x, y) = P(y)P(x | y)
• Goal: find the model h: X🡪Y, to minimize E_{P(x, y)} [loss(f(x), y)]

​

• Training data: D_tr={(x_n, y_n)} ~ P(x, y) 🡪 learn h from D_tr
• Test data: D_te={(x_m, y_m)} ~ P(x, y) 🡪evaluate sum_m (loss(h(x_m), y_m))

​

​

Mismatch between training/test data

Data collection bias

Credits: Rogerio Feris, ICCV-2019 slides

Domain adaptation

61

[Saenko al., 2019]

Domain adversarial training

​

​

​

​

62

[Credits: Hoffman 2019 ICCV tutorial]

Domain adversarial training

​

​

​

​

63

Binary classification:�SD: +1

TD: -1

[Credits: Hoffman 2019 ICCV tutorial]

Domain adversarial training

​

​

​

​

64

Binary classification:�SD: +1

TD: -1

[Credits: Hoffman 2019 ICCV tutorial]

Domain adversarial training

65

[Credits: Hoffman 2019 ICCV tutorial]

Domain adversarial training

66

The devils in the details!

Be aware of trivial solutions or poor convergence

[Credits: Hoffman 2019 ICCV tutorial]

Example results

67

[Tsai et al., CVPR 2018]

Summary of challenges

Problems:

• Limited “labeled” training data
• Mismatch between training and test data

Potential solutions:

• Transfer learning, meta learning (among data resources, modalities)
• Domain adaptation (adapt the model/training data to test data)
• Imbalanced learning (balanced, transfer among classes)
• Semi-supervised learning (leverage unlabeled data)
• Generating pseudo data
• …