�Learning with Fewer Labeled Examples��Section 19

Speaker : Hyeonmin Jang

Hanyang Univ.

7.23 Seminar

Contents

• Data augmentation

​

• Transfer Learning

-Introduction

-Adapters

-Domain Adaptation

​

• Semi-Supervised Learning

Contents

2/29

Data Augmentation

Section 19 (Introduction)

3/29

Suppose we just have a single small labeled dataset.

In some cases, we may be able to create artificially modified versions of the input vectors, while keeping the original labels unchanged.

This is called data augmentation.

Examples

Section 19 (Data Augmentation)

4/29

For image classification tasks, data augmentation method includes :

​

random crops, zooms, and mirror image flips.

If we afford to train and test the model many times using different versions of the data,

we can learn which augmentations work best, using blackbox optimization methods.

This is called AutoAugment.

Supplement

Theoretical Justification

Section 19 (Data Augmentation)

5/29

Recall

ERM(Empirical Risk Minimization) training

Theoretical Justification

Section 19 (Data Augmentation)

6/29

We can think of data augmentation as replacing the empirical distribution with the following smoothed distribution :

Theoretical Justification

Section 19 (Data Augmentation)

7/29

Putting smoothed distribution at empirical risk, we get augmented risk as :

Theoretical Justification

Section 19 (Data Augmentation)

8/29

Transfer Learning

Section 19 (Transfer Learning)

9/29

Motivation

• Many data-poor tasks have some high-level structural similarity to other data-rich tasks.

​

• For example, consider the task of fine-grained visual classification of endangered bird species.

​

Birds share many structural similarities across species.

​

First training a model on a large dataset and continuing to train it on a small dataset could produce better performance.

​

This is called transfer learning.

Section 19 (Transfer Learning)

10/29

Transfer Learning

Section 19 (Transfer Learning)

11/29

Transfer Learning

The output domains are usually different, so we need to “translate”

the output of the pre-trained model to the new domain.

Section 19 (Transfer Learning)

12/29

Transfer Learning

Model surgery

Section 19 (Transfer Learning)

13/29

Transfer Learning

Section 19 (Transfer Learning)

14/29

Transfer Learning

Section 19 (Transfer Learning)

15/29

Adapters

Disadvantage of fine-tuning all the model parameters of a pre-trained model :

​

It can be slow, since there are often many parameters, and we may need to use a small learning rate.

Keep the pre-trained model untouched, but add new parameters to modify its internal behavior to customize the feature extraction process for each task.

Alternative approach : Adapters

Section 19 (Transfer Learning)

16/29

Adapters in Transformer

Insert two MLPs inside each transformer layer, one after the multi-head attention and once after the feed-forward layers.

These MLPs have

skip connections,

so that they can be initialized to implement the identity mapping.

Section 19 (Transfer Learning)

17/29

Domain Adaptation

Motivation

Ex)

Domains might be images from a computer graphics system and real images.

Section 19 (Transfer Learning)

18/29

Domain Adaptation

We assume we do not have labeled examples from the target domain.

Goal :

​

Fit the model on the source domain, and then modify its parameters so it works on the target domain.

This is called (unsupervised) domain adaptation.

Section 19 (Transfer Learning)

19/29

Domain Adaptation

Domain adversarial learning

Train source classifier in a way that it cannot distinguish whether the input is coming from the source or target distribution.

We do this to use features that are common to both domains.

(Both distributions of source and target in feature space would be similar.)

Section 19 (Transfer Learning)

20/29

Domain Adaptation

Section 19 (Semi-supervised Learning)

21/29

Semi-Supervised Learning

Motivation

• Many applications of machine learning are in the supervised learning setting.

​

• However, it is expensive to obtain this labeled data.

​

​

• Semi-supervised learning can alleviate the need for labeled data by taking advantage of unlabeled data.

​

• Goal of semi-supervised learning is to allow model to learn the high-level structure of the data distribution from unlabeled data and only rely on the labeled data for learning the fine-grained details of a given task.

Section 19 (Semi-supervised Learning)

22/29

Semi-Supervised Learning

Section 19 (Semi-supervised Learning)

23/29

Self-Training (Pseudo-Labeling)

Use the model itself to infer predictions on unlabeled data, and then treat these predictions as labels for subsequent training.

There are two strategies :

​

• Offline self-training

​

• Online self-training

Section 19 (Semi-supervised Learning)

24/29

Self-Training (Pseudo-Labeling)

Offline Self-Training

Section 19 (Semi-supervised Learning)

25/29

Semi-Supervised Learning

Online Self-Training

Section 19 (Semi-supervised Learning)

26/29

Example

Section 19 (Semi-supervised Learning)

27/29

Semi-Supervised Learning

Basic assumption of semi-supervised learning methods is that

the decision boundary between classes should fall in a low-density region.

A good decision boundary, therefore, should not pass through clusters.

Semi-supervised learning methods make the cluster assumption.

Entropy minimization is one such method.

Section 19 (Semi-supervised Learning)

28/29

Semi-Supervised Learning

Curtain Call

29/29

Curtain Call

Thank you