1 of 32

Domain Adaptation

1

2 of 32

Domain Adaptation

  • Domain Adaptation
  • Generative Domain Adaptation
  • Adversarial Domain Adaptation
  • Discrepancy-based Domain Adaptation

2

3 of 32

Basic Assumption of AI Model

  •  

3

Train

Test

4 of 32

Lack of Generalization

  • Will the model still perform well in a deployment environment different from the training environment?

4

Train

Test

5 of 32

Mismatch Between Training and Deployment Environment

  • Phenomenon: domain shift
    • Data distribution differs between training and deployment environments
    • Equipment, sensor position, process conditions, temperature, and lighting may change
    • Learned decision criteria may no longer be valid in deployment

5

Accuracy

Train

Deploy

Training env.

Deployment env.

6 of 32

Source Domain / Target Domain

6

Source Domain

  • Existing domain used for training
  • Relatively abundant labeled data
  • Existing line / simulation / laboratory setting

Target Domain

  • New domain where the model is deployed
  • Limited or unavailable labels
  • New line / real process / changed conditions

7 of 32

Domain Shift

  • Domain shift
    • The source and target domains differ in data distribution or input-output relationship

​

  • Covariate shift
    • The input distribution P(X) changes

​

  • Examples
    • Sensor position change
    • Equipment difference
    • Changes in lighting, temperature, and vibration conditions
    • Line-specific measurement conditions

​

  • Problem
    • Source and target feature distributions may differ even for the same class
    • A decision boundary learned from the source domain may not fit target features

​

​

​

7

Source

Target

Covariate shift

8 of 32

Covariate Shift Example: Time-Varying Operating Condition

  • Difficulty acquiring diverse and high-quality data
    • Continuous changes in operating environment

​

  • Example: domain discrepancy

​

8

9 of 32

Data for Domain Adaptation: MNIST to USPS

  • Source domain: MNIST (labeled)
  • Target domain: USPS (unlabeled)
  • Task: digit classification (0 – 9)
  • Baseline Model: CNN

​

9

MNIST

USPS

10 of 32

Data for Domain Adaptation: MNIST to USPS

  • Training with MNIST (source) for Classifier

​

10

11 of 32

Performance of Baseline Classifier

  • Source-only pretrained CNN
    • MNIST Accuracy: 99.2%

11

12 of 32

Performance Degradation due to Domain Shift

  • MNIST Accuracy: 99.2%

​

​

​

​

​

​

  • USPS Accuracy: 79.47%

​

12

MNIST

(source)

USPS

(target)

13 of 32

Domain Adaptation

  • Domain Adaptation
  • Generative Domain Adaptation
  • Adversarial Domain Adaptation
  • Discrepancy-based Domain Adaptation

13

14 of 32

Generative Domain Adaptation

  • The core idea:
    • rather than adapting the model to the target domain, we adapt the data itself.
    • Source images are transformed to look like target domain images, while original labels are preserved.
    • Transformed source images are then used to train a classifier, and the resulting decision boundary is applied directly to the target domain.

14

Source

Target

Domain Adaptation

Target-like Source

Target

15 of 32

Generative Domain Adaptation: CycleGAN

  • Train CycleGAN on unpaired source and target images to learn the bidirectional mapping
  • Transform the labeled source images into the target style using GXY
  • Train a classifier on the transformed source images using the original source labels
  • Deploy the trained classifier directly on the target domain

​

15

16 of 32

Result

  • Original MNIST - the labeled source images used as input
  • Original USPS - the unlabeled target images used for style reference
  • Generated USPS - MNIST images transformed into the visual style of USPS by GXY
  • Reconstructed MNIST - the generated USPS images transformed back to the source style by GYX

​

16

Use for training

17 of 32

Domain Adaptation

  • Domain Adaptation
  • Generative Domain Adaptation
  • Adversarial Domain Adaptation
  • Discrepancy-based Domain Adaptation

17

18 of 32

Adversarial Domain Adaptation

  • The generative approach adapts the data to match the target domain at the pixel level

​

  • Adversarial domain adaptation takes a fundamentally different approach
    • rather than transforming the input images, it transforms the feature representations

​

  • Learn task-discriminative but domain-invariant features
    • A domain classifier plays a role similar to a GAN discriminator
    • The feature extractor is trained to confuse the domain classifier

​

  • The two domain are indistinguishable in the learned feature space
    • a classifier trained on labeled source features will generalize directly to target features

​

18

19 of 32

DANN: Domain-Adversarial Neural Network

  • A domain classifier plays a role similar to a GAN discriminator
  • The feature extractor is trained to confuse the domain classifier

19

Source

Target

Source data

Target data

Feature

Extractor

Source

Feature

Target

Feature

Domain

Classifier

Domain-specific feature

Correct

Fail

Domain-invariant feature

 

20 of 32

DANN: Domain-Adversarial Neural Network

  • Class-discriminative information is preserved
  • Domain-specific information is suppressed

20

Component

Role

Feature Extractor

Map source and target data into a shared feature space

Domain Classifier

Predict whether features come from source or target

Label Classifier

Learn the task using source labels

Gradient Reversal Layer

Reverse domain gradients to make features domain-invariant

Source

Target

21 of 32

DANN Example

  • Source domain: MNIST (labeled)
  • Target domain: USPS (unlabeled)
  • Task: digit classification (0 – 9)

​

21

MNIST

USPS

22 of 32

DANN Example

  • Domain adaptation using DANN

22

23 of 32

DANN Example

  • Domain adaptation using DANN

23

24 of 32

DANN Result

  • Target accuracy
    • Source-only: 80.47%
    • DANN: 93.02%

​

  • Feature Space

24

25 of 32

Domain Adaptation

  • Domain Adaptation
  • Generative Domain Adaptation
  • Adversarial Domain Adaptation
  • Discrepancy-based Domain Adaptation

25

26 of 32

Discrepancy-based DA: CORAL (Correlation Alignment)

  •  

26

27 of 32

Discrepancy-based DA: CORAL (Correlation Alignment)

  • To find a linear transformation A aligning source covariance to target covariance

27

 

Source

Target

Source

Target

 

28 of 32

DeepCORAL

  • DeepCORAL extends this idea by integrating the CORAL objective directly into neural network training
  • Allowing the feature representations to be learned with reduced covariance discrepancy
    • Use source labels for classification loss
    • Matching the covariance of the two feature distributions�

​

28

Source data

Target data

Feature

Extractor

Source

Feature

Target

Feature

 

Classifier

 

29 of 32

DeepCORAL

  •  

29

 

Source Feature

Target Feature

30 of 32

DeepCORAL Example

  • Pretraining with MNIST (source)

​

30

31 of 32

DeepCORAL Example

  • Domain adaptation using DeepCORAL

​

​

​

​

​

​

​

31

 

 

32 of 32

DeepCORAL Result

  • Target accuracy
    • Source-only: 79.47%
    • DeepCORAL: 87.02%

32