1 of 38

Self-supervised Learning

Prof. Seungchul Lee

Industrial AI Lab.

2 of 38

Supervised Learning and Transfer Learning

  • Supervised pretraining on large labeled datasets has led to successful transfer learning
    • ImageNet

​

​

​

​

​

​

    • Pretrain for fine-grained image classification of 1000 classes
    • Use feature representations for downstream tasks, e.g., object detection, image segmentation, and action recognition

2

3 of 38

Knowledge Transfer

  • Feature learning in pretext task

​

  • Wide variety of downstream tasks
    • Classification
    • Regression
    • Object detection
    • Segmentation

3

4 of 38

Supervised Learning

  • But supervised pretraining comes at a cost…
    • Time-consuming and expensive to label datasets for new tasks

​

    • Domain expertise needed for specialized tasks
      • Radiologists to label medical images
      • Native speakers or language specialists for labeling text in different languages

​

  • To relieve the burden of labelling,
    • Semi-supervised learning
    • Weakly-supervised learning
    • Unsupervised learning

4

5 of 38

Self-supervised Learning

  • Self-supervised learning (SSL): supervise using labels generated from the data without any manual or weak label sources
    • Sub-class of unsupervised learning

​

  • Idea: Hide or modify part of the input. Ask model to recover input or classify what changed
    • Self-supervised task referred to as the pretext task can be formulated using only unlabeled data
    • The features obtained from pretext tasks are transferred to downstream tasks like classification, object detection, and segmentation

5

6 of 38

Self-supervised Pretext Tasks

  1. Solving the pretext tasks allow the model to learn good features.
  2. We can automatically generate labels for the pretext tasks.

6

Image from http://cs231n.stanford.edu/

image completion

rotation prediction

“jigsaw puzzle”

colorization

7 of 38

Pretext Task – Context Prediction

  • After creating 9 patches from one input image, the classifier is trained on the location information between the middle and other patches

​

7

Carl Doersch, Abhinav Gupta, Alexei A. Efros, 2015, "Unsupervised Visual Representation Learning by Context Prediction," Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1422-1430.

8 of 38

Pretext Task – Context Prediction

  • After creating 9 patches from one input image, the classifier is trained on the location information between the middle and other patches

​

  • A pair of middle patch and other patch is given as the input for the network
  • Method to avoid trivial solutions
    • uneven spacing between patches

​

8

Carl Doersch, Abhinav Gupta, Alexei A. Efros, 2015, "Unsupervised Visual Representation Learning by Context Prediction," Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1422-1430.

9 of 38

Pretext Task – Jigsaw Puzzle

  • Generate 9 patches from the input image
  • After shuffling the patches, learn a classifier that predicts permutations to return to the original position

​

9

Noroozi, M., and Favaro, P., 2016, "Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles," Computer Vision – ECCV 2016, 69–84.

10 of 38

Pretext Task – Jigsaw Puzzle

  • Generate 9 patches from the input image
  • After shuffling the patches, learn a classifier that predicts permutations to return to the original position

​

​

10

Noroozi, M., and Favaro, P., 2016, "Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles," Computer Vision – ECCV 2016, 69–84.

11 of 38

Pretext Task – Image Colorization

  • Given a grayscale photograph as input, image colorization attacks the problem of hallucinating a plausible color version of the photograph
  • Transfer the trained encoder to the downstream task

11

Zhang, R., Isola, P., and Efros, A. A., 2016, "Colorful Image Colorization," Computer Vision – ECCV 2016, 649–666.

12 of 38

Pretext Task – Image Colorization

  • Given a grayscale photograph as input, image colorization attacks the problem of hallucinating a plausible color version of the photograph
  • Transfer the trained encoder to the downstream task

​

  • Training data generation for self-supervised learning

12

Image from https://amitness.com/2020/02/illustrated-self-supervised-learning/

Grayscale filter

​

13 of 38

Pretext Task – Image Colorization

  • Given a grayscale photograph as input, image colorization attacks the problem of hallucinating a plausible color version of the photograph
  • Transfer the trained encoder to the downstream task

​

  • Network architecture

13

Image from https://amitness.com/2020/02/illustrated-self-supervised-learning/

14 of 38

Pretext Task – Image Super-resolution

  • What if we prepared training pairs of (small, upscaled) images by downsampling millions of images we have freely available?

​

  • Training data generation for self-supervised learning

​

14

Make it smaller (downsampling)

Image from https://amitness.com/2020/02/illustrated-self-supervised-learning/

15 of 38

Pretext Task – Image Super-resolution

  • What if we prepared training pairs of (small, upscaled) images by downsampling millions of images we have freely available?

​

  • Network architecture

​

​

15

Image from https://amitness.com/2020/02/illustrated-self-supervised-learning/

16 of 38

Pretext Task – Image Inpainting

  • What if we prepared training pairs of (corrupted, fixed) images by randomly removing part of images?

​

  • Training data generation for self-supervised learning

​

16

Random missing region

Image from https://amitness.com/2020/02/illustrated-self-supervised-learning/

17 of 38

Pretext Task – Image Inpainting

  • What if we prepared training pairs of (corrupted, fixed) images by randomly removing part of images?

​

  • Network architecture

​

​

17

Image from https://amitness.com/2020/02/illustrated-self-supervised-learning/

18 of 38

Deep Artificial Neural Networks

  • Complex/Nonlinear universal function approximator
    • Linearly connected networks
    • Simple nonlinear neurons

​

​

​

​

18

Class 1

Class 2

…

…

…

…

…

…

nonlinear

linear

Feature learning

Classification

Output

Input

19 of 38

Benefits of Self-supervised Learning

  • Like supervised pretraining, can learn general-purpose feature representations for downstream tasks

​

  • Reduce expense of hand-labeling large datasets

​

  • Can leverage nearly unlimited unlabeled data available on the web

19

20 of 38

Pipeline of Self-supervised Learning

  1. Within pretext tasks, deep neural network learns visual features of input unlabeled data

​

  • The learned parameters of the network remain fixed and the trained network serves as a pre-trained model for downstream tasks

​

  • The pre-trained model is transferred to downstream tasks and is fine-tuned

​

  • The performance of downstream tasks is used to evaluate the methodology used in pretext tasks to learn features from unlabeled data

​

20

Jing, L., & Tian, Y., 2021, "Self-supervised visual feature learning with Deep Neural Networks: A survey," IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11), 4037–4058.

21 of 38

Downstream Tasks

  • After transferring the neural network pre-trained by the pretext task, freeze the weights and build additional layers for the downstream tasks

​

  • Wide variety of downstream tasks
    • Classification
    • Regression
    • Object detection
    • Segmentation

21

22 of 38

Pretext Task – Rotation

  • Hypothesis: a model could recognize the correct rotation of an object only if it has the “visual commonsense” of what the object should look like
    • Self-supervised learning by rotating the entire input images
    • The model learns to predict which rotation is applied (4-way classification)

22

Spyros Gidaris, Praveer Singh, & Nikos Komodakis, 2018, "Unsupervised Representation Learning by Predicting Image Rotations," Le Centre Pour La Communication Scientifique Directe - HAL - Inria (2018).

23 of 38

Self-supervised Learning with RotNet in TensorFlow

23

24 of 38

Input Data Summary and Key Concept

  • Pretext Task
    • Input data: 10,000 MNIST images without labels

​

  • Downstream Task and Supervised Learning (for performance comparison)
    • Training data: 1,000 MNIST images with labels
    • Test data: 300 MNIST images with labels

​

  • Key concepts
    • For transfer learning, we used to train networks like VGG 16 with large image dataset with labels such as ImageNet
    • With self-supervised learning, we train such networks with unlabeled image datasets which have larger number of data than labeled image datasets have and perform transfer learning
    • Comparing downstream task performance with that of supervised learning is equal to comparing the performance of (self-supervised) transfer learning and supervised learning performance

​

​

24

25 of 38

Pretext Task – Input Image Rotation

  • Need to generate rotated images and their labels to train the model for pretext task
    • 0 degree rotation: [1, 0, 0, 0]
    • 90 degree rotation: [0, 1, 0, 0]
    • 180 degree rotation: [0, 0, 1, 0]
    • 270 degree rotation: [0, 0, 0, 1]

25

26 of 38

Pretext Task – Build CNN Model

  • Convolution Neural Network model for rotation classification
    • Model: three convolution layers and one fully connected layer
    • Optimizer: adam

26

27 of 38

Pretext Task – Training

  • Training the model for the pretext task

​

​

​

​

​

​

​

​

27

28 of 38

Downstream Task – Training and Result

  • Training the model for the downstream task
  • Freezing trained parameters to transfer them for the downstream task

​

28

29 of 38

Downstream Task – MNIST Image Classification

  • Model: two convolution layers and one fully connected layer
    • Two convolution layers are transferred from the model for the pretext task
    • Single fully connected layer is trained only

29

30 of 38

Supervised Learning for Comparision

  • Convolution Neural Networks for MNIST image classification
    • Model: Same model architecture with the model for the downstream task
    • The number of total parameter is the same with the model for the downstream task, but is has zero non-trainable parameters

30

31 of 38

Self-supervised Learning vs. Supervised Learning

  • Self-supervised learning showed higher accuracy in classifying the MNIST test images

31

32 of 38

Self-supervised Learning in LLM

32

33 of 38

Language Modeling

  • Given sequence of words so far (context), predict what comes next?

​

​

33

34 of 38

Language Modeling

  • Given sequence of words so far (context), predict what comes next?

​

​

34

35 of 38

Recipe for Creating a Language Model

  • Step 1: Collect a very large data:
    • Wikipedia, StackOverflow
    • Quora, Public Social media
    • Github, Reddit

​

  • Step 2: Ask LM to predict the next word in a sentence
    • Randomly truncate last part of input sentence
    • Calculates probabilities of missing words
    • Adjust and feed back to the model to match the ground truth

​

  • Step 3: Repeat over the whole corpus

35

36 of 38

Self-supervised Learning

36

37 of 38

Exploiting a Pre-trained Model: Fine-Tuning

37

Foundation Model

38 of 38

The Second Kind of Label: Comparisons

  • It is often much easier to compare answers instead of writing answers

38

LLM by SSL + Reinforcement Learning