1 of 57

Dog Emotion Prediction Based On Transfer Learning

Team: Tree New Bee

Wei Shan, Jeff Zhuo, Xi Du, Tianrui Ye, Yuewei Wang

2 of 57

Introduction

  • Motivation
    • We love Dogs
  • Objective
    • Image Classification
      • 4 classes
  • Background
    • Transfer Learning

3 of 57

Introduction

  • Motivation
    • We love Dogs
  • Objective
    • Emotion Classification
      • 4 classes
  • Background
    • Transfer Learning

Angry

Happy

Relaxed

Sad

4 of 57

Introduction

  • Motivation
    • We love Dogs
  • Objective
    • Image Classification
      • 4 classes
  • Background
    • Transfer Learning

5 of 57

Introduction(Background - Transfer Learning)

Image Credit: Prof. Li

Task we want to accomplish

Task already accomplished

6 of 57

Introduction(Background - Transfer Learning)

  • Task A and B have similar input x.
  • You have a lot more data for Task A than Task B.
  • Low level features from A could be helpful for learning B.

Works cited: Dr. Andrew Ng, https://www.youtube.com/watch?v=yofjFQddwHE

Image Credit: Prof. Li

7 of 57

Introduction(Background - Transfer Learning)

  • Model trained by big dataset?

Image Credit: Prof. Li

8 of 57

Introduction(Background - Transfer Learning)

  • Model trained by big dataset?
  • How to transfer knowledge?

Image Credit: Prof. Li

9 of 57

Introduction(Background - Transfer Learning)

Q: Model trained by big dataset?

​

Image Credit: https://www.analyticssteps.com/blogs/how-transfer-learning-done-neural-networks-and-convolutional-neural-networks

10 of 57

Introduction(Background - Transfer Learning)

Q: Model trained by big dataset?

A: Pre-trained Model!

    • ResNet
    • VGG
    • AlexNet
    • MobileNet
    • GoogLeNet

​

Image Credit: https://www.analyticssteps.com/blogs/how-transfer-learning-done-neural-networks-and-convolutional-neural-networks

11 of 57

Introduction(Background - Transfer Learning)

Q: Model trained by big dataset?

A: Pre-trained Model!

    • ResNet
    • VGG
    • AlexNet
    • MobileNet
    • GoogLeNet

Q: How to transfer knowledge?

​

Image Credit: https://www.analyticssteps.com/blogs/how-transfer-learning-done-neural-networks-and-convolutional-neural-networks

12 of 57

Introduction(Background - Transfer Learning)

Q: Model trained by big dataset?

A: Pre-trained Model!

    • ResNet
    • VGG
    • AlexNet
    • MobileNet
    • GoogLeNet

Q: How to transfer knowledge?

​

Image Credit: https://www.analyticssteps.com/blogs/how-transfer-learning-done-neural-networks-and-convolutional-neural-networks

13 of 57

Introduction(Background - Transfer Learning)

Q: Model trained by big dataset?

A: Pre-trained Model!

    • ResNet
    • VGG
    • AlexNet
    • MobileNet
    • GoogLeNet

Q: How to transfer knowledge?

A: Freeze convolutional layers’ parameters!

​

Image Credit: https://www.analyticssteps.com/blogs/how-transfer-learning-done-neural-networks-and-convolutional-neural-networks

14 of 57

Introduction(Background - Transfer Learning)

Q: Model trained by big dataset?

A: Pre-trained Model!

    • ResNet
    • VGG
    • AlexNet
    • MobileNet
    • GoogLeNet

Q: How to transfer knowledge?

A: Freeze convolutional layers’ parameters!

​

Image Credit: https://www.analyticssteps.com/blogs/how-transfer-learning-done-neural-networks-and-convolutional-neural-networks

15 of 57

Review of Literature

16 of 57

Review of Literature: Image-based Sentiment Analysis

  • Lei Zhang, Shuai Wang, & Bing Liu. (2018). Deep Learning for Sentiment Analysis : A Survey.
  • Mittal, N., Sharma, D., & Joshi, M. (2018). Image Sentiment Analysis Using Deep Learning. In 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI) (pp. 684-687).
  • Cetinic, E., Lipic, T., & Grgic, S. (2019). A Deep Learning Perspective on Beauty, Sentiment, and Remembrance of Art. IEEE Access, 7, 73694-73710.

​

17 of 57

Review of Literature: Pre-trained Models

  • Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. (2015). Deep Residual Learning for Image Recognition.
  • Karen Simonyan, & Andrew Zisserman. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition.
  • Krizhevsky, A., Sutskever, I. & Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks.
  • Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, & Hartwig Adam. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications.
  • Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, & Andrew Rabinovich. (2014). Going Deeper with Convolutions.

18 of 57

Review of Literature: Transfer Learning

  • Pan, S., Tsang, I., Kwok, J., & Yang, Q. (2011). Domain Adaptation via Transfer Component Analysis. IEEE Transactions on Neural Networks, 22(2), 199-210.
  • Jason Yosinski, Jeff Clune, Yoshua Bengio, & Hod Lipson. (2014). How transferable are features in deep neural networks?
  • Y. Zhang and Q. Yang. A Survey on Multi-Task Learning.

19 of 57

Methodology (Pre-trained Model Selection)

​

  • Pre-trained Model
    • ResNet
    • VGG
    • AlexNet
    • MobileNet
    • GoogLeNet

20 of 57

Methodology (Pre-trained Model Selection)

​

  • Pre-trained Model
    • ResNet
    • VGG
    • AlexNet
    • MobileNet
    • GoogLeNet

More conv …

21 of 57

Methodology (Pre-trained Model Selection)

  • Pre-trained Model
    • ResNet
    • VGG
    • AlexNet
    • MobileNet
    • GoogLeNet

More conv …

Parameters Freezed

22 of 57

Methodology (Pre-trained Model Selection)

  • Pre-trained Model
    • ResNet
    • VGG
    • AlexNet
    • MobileNet
    • GoogLeNet
  • Change the last fully connected (fc) layer
    • Output size: 1000 -> 4

More conv …

fc 4

Parameters Freezed

23 of 57

Methodology (Pre-trained Model Selection)

​

  • Pre-trained Model
    • ResNet
    • VGG
    • AlexNet
    • MobileNet
    • GoogLeNet
  • Change the last fully connected (fc) layer
    • Output size: 1000 -> 4
  • Accuracy Comparison

More conv …

fc 4

Parameters Freezed

24 of 57

Result (Pre-trained Model Selection)

Model

ResNet18

ResNet50

VGG16

AlexNet

MobileNet

GoogLeNet

Training Accuracy

44%

54%

58%

47%

51%

46%

25 of 57

Result (Pre-trained Model Selection)

Model

ResNet18

ResNet50

VGG16

AlexNet

MobileNet

GoogLeNet

Training Accuracy

44%

54%

58%

47%

51%

46%

26 of 57

Result (Pre-trained Model Selection)

Model

ResNet18

ResNet50

VGG16

AlexNet

MobileNet

GoogLeNet

Training Accuracy

44%

54%

58%

47%

51%

46%

Image Credit: http://jaree.its.ac.id/index.php/jaree/article/view/191

27 of 57

Result (Pre-trained Model Selection)

Model

ResNet18

ResNet50

VGG16

AlexNet

MobileNet

GoogLeNet

Training Accuracy

44%

54%

58%

47%

51%

46%

Image Credit: http://jaree.its.ac.id/index.php/jaree/article/view/191

28 of 57

Methodology

  • Classifier Modification
  • Multi-Task Learning
  • Fine-tuning
  • Unsupervised Domain Adaptation

29 of 57

Methodology

  • Classifier Modification
  • Multi Task Learning
  • Fine-tuning
  • Unsupervised Domain Adaptation

30 of 57

Methodology (Classifier Modification)

  • Fully Connected Layer

Parameters Freezed

fc 4

31 of 57

Methodology (Classifier Modification)

  • Fully Connected Layer
  • Bayes Classifier (LDA/ QDA)
  • KNN
  • SVM
  • Random Forest

Parameters Freezed

32 of 57

Methodology (Classifier Modification)

Bayes Classifier (LDA/ QDA)

Parameters Freezed

33 of 57

Methodology (Classifier Modification)

KNN

Parameters Freezed

34 of 57

Methodology (Classifier Modification)

SVM

Parameters Freezed

35 of 57

Methodology (Classifier Modification)

Random Forest

Parameters Freezed

36 of 57

Result (Classifier Modification)

Model

FC

LDA

QDA

KNN

SVM

Random Forest

Training Accuracy

54%

79%

82%

48%

79%

61%

37 of 57

Result (Classifier Modification)

Model

FC

LDA

QDA

KNN

SVM

Random Forest

Training Accuracy

54%

79%

82%

48%

79%

61%

Testing

Accuracy

53%

31%

35%

20%

59%

27%

38 of 57

Methodology

  • Classifier Modification
  • Multi Task Learning
  • Fine-tuning
  • Unsupervised Domain Adaptation

39 of 57

Methodology (Multi Task Learning)

Image Credit: I made them

Input 1

Task 1

Input 2

Task 2

Single Task Learning

Input 1

Task 1

Input 2

Task 2

Multi Task Learning

Conv

Layers

FC

Layers

Conv

Layers

FC

Layers

40 of 57

Methodology (Multi Task Learning)

  • Tasks should be related.

​

Image Credit: I made them

Works Cited: Y. Zhang and Q. Yang, "A Survey on Multi-Task Learning,"

Input 1

Task 1

Input 2

Task 2

Single Task Learning

Input 1

Task 1

Input 2

Task 2

Multi Task Learning

Conv

Layers

FC

Layers

Conv

Layers

FC

Layers

41 of 57

Methodology (Multi Task Learning)

Image Credit: I made them

Input 1

Task 1

Input 2

Task 2

Conv

Layers

FC

Layers

42 of 57

Methodology (Multi Task Learning)

Image Credit: I made them

Task 2

Conv

Layers

FC

Layers

Positive

Positive

Negative

Negative

Input 2

43 of 57

Methodology (Multi Task Learning)

Image Credit: I made them

Conv

Layers

FC

Layers

Relaxed

Happy

Sad

Angry

Positive

Positive

Negative

Negative

44 of 57

Methodology (Multi Task Learning)

Image Credit: I made them

ResNet50

(pre-trained model)

FC

Layers

Relaxed

Happy

Sad

Angry

Positive

Positive

Negative

Negative

45 of 57

Methodology (Multi Task Learning)

Image Credit: I made them

ResNet50

(pre-trained model)

FC

Layers

Relaxed

Happy

Sad

Angry

Positive

Positive

Negative

Negative

……

2048

……

46 of 57

Result (Multi Task Learning)

  • 35%

47 of 57

Methodology

  • Classifier Modification
  • Multi Task Learning
  • Fine-tuning
  • Unsupervised Domain Adaptation

48 of 57

Methodology (Fine-Tuning)

Big Idea: Low-Level features are general and High-level features are more specific.

  • Low-Level features learnt are transferable
  • Higher Layers capture features specialized to the original training tasks

​

Figure 1: Hierarchy of Features in CNNs

49 of 57

Methodology (Fine-Tuning)

ResNet50

  • Previously: freeze all feature layers
    • Train only classification layer

Image Credit: ResNet50 Architecture

Train

Freeze

FC 4

50 of 57

Methodology (Fine-Tuning)

ResNet50

  • Previously: freeze all feature layers
    • Train only classification layer
  • Higher layer may capture features that are specialized for ImageNet task
    • Train last Feature Layer & FC Layer
  • Use initial pre-train weights leads to better performance and generalization (Yosinski)

Image Credit: ResNet50 Architecture

Train

Freeze

Train

Freeze

FC 4

Jason Yosinski, Jeff Clune, Yoshua Bengio, & Hod Lipson. (2014). How transferable are features in deep neural networks?

51 of 57

Results (Fine-Tuning)

Method

Training on FC Layer

Training on FC + Stage 4

Accuracy Increase

Training Accuracy

54.5%

63.8%

9.3%

Test Accuracy

53.3%

57.6%

4.3%

Accuracy Difference

-1.2%

-6.2%

​

52 of 57

Methodology

  • Classifier Modification
  • Multi Task Learning
  • Fine-tuning
  • Unsupervised Domain Adaptation

53 of 57

Methodology (Unsupervised Domain Adaptation)

What is Domain Adaptation

  • Source domain → target domain
  • Two features: brightness, raindrop
  • Bright + No raindrop = Sunny
  • Dark + Rain drop = Rainy
  • Dark + No raindrop = ?
  • simple/single knowledge → complex situation

Why Domain Adaptation useful

  • Pre-trained model is trained on a specific dataset
  • Our dataset contains real world pictures

54 of 57

Methodology (Unsupervised Domain Adaptation)

55 of 57

Methodology (Unsupervised Domain Adaptation)

56 of 57

Results(Unsupervised Domain Adaptation)

ResNet50 Accuracy

Training Accuracy: 62.8%

Testing Accuracy: 53%

Training Accuracy Increase: 9.8%

​

57 of 57

Conclusion

  • Introduced Pre-train Models and Transfer Learning
  • Selected best Pre-train Model
  • Classifier Modification
  • Multi-task Learning
  • Fine-Tuning
  • Unsupervised Domain Adaptation

​