Weakly Supervised Segmentation

Jike Zhong, Wenjin Fu, Tianle Chen

What is Weakly Supervision

• Incomplete Supervision

​

• Inexact Supervision

​

• Inaccurate Supervision

​

Incomplete

Only a (usually small) subset of training data is given with labels while the other data remain unlabeled

• There are two major techniques for this purpose
• Active learning
• Semi-supervised learning
1. generative methods
2. graph-based methods
3. low-density separation methods
4. disagreement-based methods.

​

Inaccurate Supervision

Inaccurate supervision concerns the situation in which the supervision information is not always ground-truth

• Scenario is learning with label noise
• Scenario of inaccurate supervision occurs with crowdsourcing

​

Inexact Supervision

Inexact supervision concerns the situation in which some supervision information is given, but not as exact as desired

​

• Typical scenario is when only coarse-grained label information is available.
• Detection / Segmentation with only image level label

​

Weakly Supervised Semantic Segmentation (WSSS)

• Task of semantic segmentation
• Challenges of this task (why we need WSSS)
• Task of WSSS

Different Approach on WSSS

​

• CNN-Base Approach
• FCN-Base Approach
• Structural causal
• GAN-Base Approach
• Coseg-Base Approach
• CAM-Base Approach

Learning Deep Features for Discriminative Localization��Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, Antonio Torralba

CVPR 2016

Motivation

Motivation & CNN Review

Architecture: CAM

Last layer of conv

softmax

Architecture: CAM

→ activation of unit k in the last convolutional layer

→ global average pooling (GAP)

→ input to the softmax for a given class c

→ weight corresponding to class c for unit k

(importance of F_k for class c)

→ output of the softmax for class c

Class activation map (CAM)

Last layer of conv

softmax

Results

Comparison of the localization from GoogleNet-Gap (uppper two) and the backpropagation using AlexNet (lower two)

Applications

Weakly-Supervised Object Detection

Weakly-Supervised text detector

Visual Question Answering

�Weakly-supervised on non-CAM base method: Co-segmentation

�

Deep Object Co-segmentation via Spatial-Semantic Network Modulation

�Kaihua Zhang, Jin Chen, Bo Liu, Qingshan

AAAI 2019

Beyond CAM - Puzzle CAM:

Motivation

• Traditional CAM only focuses on the most discriminative area.
• Tiled images (b) produce different CAMS compared to whole image (a).
• Motivation: Adapting an Attention-Based feature learning method (right).

Puzzle CAM: Overall Architecture

• Puzzle Module + Reconstructing Regularization

Puzzle CAM: Puzzle Module

• Generic CAM:

A = Σ_c(argmax (Ac = θc⊤f ))

(θc: weights of the c-channel classifier; f = F(I): feature map)

• Puzzle CAM:

I -> {I^1,1, I^1,2, I^2,1, I^2,2} (I is the input image)

Obtain all A^i,j for each I^i,j

Merge A^i,j into single A

​

Puzzle CAM: Loss Design

• 2 classification losses, one for original

image, one for reconstructed image

(area estimation)

​

​

• 1 reconstructing regularization (narrow

the gaps between the pixel- and

image-level supervision processes)

​

• Overall loss

​

Puzzle CAM - Enhancement: Affinity Net

• Left: Semantic affinity between two features
• Right: Architecture of affinity net

​

​

​

Puzzle CAM: Pseudo Mask Quality

• Pseudo semantic segmentation labels in mIoU, evaluated on the PASCAL VOC 2012 training set. RW, random walk with AffinityNet; dCRF, dense conditional random field

​

​

Puzzle CAM: Results

• Obtained on PASCAL VOC 2012 Val set

​

​

​

​

​

​

​

Original images

Ground Truth

Res from Puzzle-CAM

Puzzle CAM: Results

• Obtained on PASCAL VOC 2012 Val set
• Comparing to State-Of-The-Art

​

​

​

​

​

​

​

Extended CAM-Based Work

Dilated convolution

Multi-layer feature fusion

Saliency-guided iterative training

Stochastic feature selection

…

Extended CAM-Based Work

Dilated convolution

Multi-layer feature fusion

Saliency-guided iterative training

Stochastic feature selection

…

Extended CAM-Based Work

Dilated convolution

Multi-layer feature fusion

Saliency-guided iterative training

Stochastic feature selection

…

Extended CAM-Based Work

Dilated convolution

Multi-layer feature fusion

Saliency-guided iterative training

Stochastic feature selection

…

Extended Work

CIAN: Cross-Image Affinity Net for Weakly Supervised Semantic Segmentation

​