1 of 18

Multi-Anchor Active Domain Adaptation for Semantic Segmentation

-- Munan Ning

腾讯天衍实验室

TENCENT JARVIS LAB

2 of 18

Contents

Problem definition
Related work
Method
Result

Tencent Jarvis

3 of 18

Problem definition

Domain bias for semantic segmentation

Problem：The prediction distortion in when adapting models from Source domain to Target domain

Reasons:

1) Appearance bias: Source and target domain have different image styles

2) Content bias: Source and target domain vary in content distribution

Essence: Lacking of target information

Ground truth

Adaptation result

Target sample

Tencent Jarvis

At first, we want to take a look at the domain bias problem in semantic segmentation task.

If we adapt a well-trained model in the source domain to the target domain directly, a severe prediction distortion will happen. Just like shown in the figures, a model trained on GTA-5 dataset is deployed on the Cityscapes dataset, and output an error-prone segmentation result.

There are two main reasons for domain bias. The first one is appearance bias, which means source and target domains have different image styles. The second is content bias, which means the source and target domains vary in content distribution. Such a bias is often ignored in UDA methods because they suppose the source and target domains share similar content.

The essence of domain bias can be summarized as lacking of target information, which will be discussed in more detail as the following page.

4 of 18

Problem definition

Visualization of two biases

Appearance bias: Source and target domain have different image styles
Content bias: Source and target domain vary in content distribution

Summary:

1) Possessing different image styles

2) Sharing similar content structure but not same

Source-1（GTA5）

Source-2（SYNTHIA）

Target (Cityscapes)

Image

Label

Tencent Jarvis

5 of 18

Related work-UDA methods

Unsupervised Domain Adaptation for semantic segmentation

Definition：Aligning the target-domain distribution towards the source-domain distribution

AdaptSegNet[1]:

A typical adversarial based UDA method:

Compose of a Generator (shared segmentation model) and a Discriminator

Aligning features and predictions from source domain and target domain

[1] Tsai, Yi-Hsuan, et al. "Learning to adapt structured output space for semantic segmentation." CVPR. 2018.

Tencent Jarvis

Before introducing our framework, let’s briefly introduce the existing unsupervised domain adaptation approaches.

Most unsupervised domain adaptation segmentation methods are aligning the garget-domain distribution towards the source-domain distribution with adversarial learning.

The AdaptSegNet is the most typical UDA method. It consider the segmentation network as a generator, and introduce an additional discriminator to distinguish whether the segmentation map comes from source or target domain. By encouraging the generator cheating the discriminator with an adversarial learning, the features and predictions from source and target domains are aligned. As a result, the target predictions are aligned to the source predictions.

In another word, model can generate source-like predictions from target samples.

6 of 18

Related work-UDA methods

More UDA methods

DISE[2] proposed a disentangled representation learning architecture to preserve structural information during image translation

Feature aligning methods such as CLAN [3] and CAG [4] utilized category-based distribution alignment to adapt the source and target domains in the feature and output spaces

[2] Chang, Wei-Lun, et al. "All about structure: Adapting structural information across domains for boosting semantic segmentation." CVPR. 2019.

DISE:

CLAN:

CAG:

[3] Luo, Yawei, et al. "Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation." CVPR. 2019.

[4] ZHANG, Qiming, et al. "Category Anchor-Guided Unsupervised Domain Adaptation for Semantic Segmentation." NIPS. 2019.

Tencent Jarvis

There are more methods are employed to improve the performance of domain adaptation.

Based on the assumption that the structure information is domain-invariant, the DISE proposed a disentangled representation learning architecture to preserve structural information during image translation. To be more detailed, it forces the models to learn the texture information of source and target textures, and extract content information from both source and target images.

The AdaptSegNet employs a simple global alignment for source and target images. While different categories locate in different locations in the street-view images. For example, Cars Can’t Fly up in the Sky. So the CLAN and CAG propose the category-based distribution alignment, try to align the distribution per category. For example, aligning cars to cars, roads to roads. These approach can effectively make the prediction more reasonable.

7 of 18

Related work-Active learning

Active learning for single domain

Definition：Optimizing performance at a low annotation cost, by actively selecting most informative samples

Labeling informative samples (right) perform better than labeling random samples (middle)

Tencent Jarvis

8 of 18

Related work-Active learning

Active learning for domain adaptation

AADA (Active Adversarial Domain Adaptation) [5]：Exploring a combination between two related problems: adversarial domain alignment and importance sampling for adapting models across domains.

Start from an unsupervised domain adaptation setting, select samples using importance weight and re-train the model.

[5] Su, Jong-Chyi, et al. "Active adversarial domain adaptation." WACV. 2020.

Tencent Jarvis

9 of 18

Method-Motivation

Visualization of distribution distortion

The average latent representation of source domain (blue squares) and target domain (yellow squares) share little overlap (region ①) along with large discrepancy (regions ② and ③)

In region ①, source and target samples share similar content, and Adv. methods [1] achieve good performance

In region ② and ③, inferior segmentations are yielded because of obvious distribution distortion

Tencent Jarvis

Now we can introduce our method. We aim to solve two problem. First, UDA methods suffer a severe distribution distortion because they lack the target-specific information. Second, the existing active domain adaptation methods utilize course global active metrics.

In this page we visualize the distribution distortion. We utilize a T-SNE to show the distribution of latent representations extracted from images. To be more detailed, the blue squares and yellow squares denote the average of latent representation of source and target domain respectively, while the red and green dots denote the .

There are three regions, the region 1 denotes where source and target samples share similar content, and Adv. methods [1] achieve good performance.

While for other samples, target samples are forced aligned to the source mean, just as the region 2. For other target-specific samples, Adv method can not effectively align them (red dots in region 3). The corresponding segmentation proves the wrong alignment can cause inferior segmentations.

10 of 18

Method-Framework

Solution

Essence:

Multiple anchoring mechanism

Two phases:

①Active target sample selection against source anchors (part. A)

②Semi-supservised domain adapt-ation (part. B-D)

Tencent Jarvis

11 of 18

Method-Framework

How:

①Calculate the features of each sample

②Cluster sample features with K-means

③ Denote the cluster centers as anchors

Tencent Jarvis

12 of 18

Method-Framework

Active sample selection

Purpose:

①Measure distance between the target-domain samples and the source-domain anchors.

②Find the most far away samples.

Cross-domain metric:

Calculate the distance of target sample to the nearest anchor as the sample-to-domain distance

Tencent Jarvis

13 of 18

Method-Framework

Semi-supervised domain adaptation

Loss functions:

①Segmentation loss for labeled source and active samples

②Soft alignment loss between target sample and target anchors

③Pseudo label segmentation loss for unlabeled target samples

① Segmentation loss

② Soft alignment loss

③Pseudo label segmentation loss

Tencent Jarvis

Based on the labeled active samples, we employ the semi-supervised domain adaptation training strategy.

At first, for all the labeled source and active samples, a cross-entropy based segmentation loss is employed to provide direct supervision.

Besides, a soft alignment loss is introduced to close the target samples and corresponding anchors. Because only few samples are labeled, such an alignment loss can effectively avoid generating predictions with seriously error. Of course, the target anchors are re-calculated from target samples to model the target distribution.

At last , the pseudo label segmentation loss is employed. The soft alignment loss may guide prediction closing to a trivial solution, while the pseudo label can work as a self-sharpness function, reserving more individual information for each image.

14 of 18

Result

Comparison with Sotas

We observe substantial improvements over the UDA methods, suggesting that with carefully selected active samples, little manual annotation workload can lead to large performance
The proposed method outperforms another active DA method, i.e., AADA, by a large margin, demonstrating the effectiveness of the proposed multi-anchor strategy

Tencent Jarvis

15 of 18

Result

Comparison with Sotas

We observe substantial improvements over the UDA methods, suggesting that with carefully selected active samples, little manual annotation workload can lead to large performance
The proposed method outperforms another active DA method, i.e., AADA, by a large margin, demonstrating the effectiveness of the proposed multi-anchor strategy

Tencent Jarvis

16 of 18

Result

Comparison with active metrics

We can observe that the proposed multi-anchor strategy delivers the best segmentation performance in mIoU, suggesting that better active samples are selected by our proposed strategy.

Tencent Jarvis

17 of 18

Result

Visualization

By alleviating the distortion of target features, fewer segmentation errors as well as more precise boundaries can be obtained with the proposed MADA method.

Tencent Jarvis

18 of 18

Thanks