1 of 34

Neural Style Transfer

Eslam Mahmoud Abd El-Aziim

Moaz Mohammed Elsayed Ammar

Nour El-Deen Abd El-Hameed

​

Supervised By

​

Dr. Sameh Basha

Department of Mathematics

Faculty of Science

Cairo University

2 of 34

Agenda

  • Introduction
  • Motivation
  • Problem Statement
  • Previous Work
  • Proposed Model
  • Results and Discussion
  • Conclusion
  • Other Challenges
  • Future Work

3 of 34

Introduction

  • Style transfer is considered a generalized problem of texture synthesis.
  • Changing the style of an image requires redrawing the image from scratch manually.
  • Redrawing images takes time and effort.
  • It needs professional artistic skills.

4 of 34

Introduction

  • Gatys et al had a novel approach using Convolutional Neural Networks (CNNs).
  • Their approach separate the content and style of the images using VGG convolution network.
  • They formulate an optimization problem that searches for new image that has similar content image and style image.
  • These methods were not suitable for photorealistic style transfer because they are adding too much stylization and strong abstraction which is not preferred in photo realistic.

5 of 34

Types of Neural Style Transfer

  • Artistic
    • Makes an abstraction of the content image which may change the structure of the content image according to the style image
  • Photorealistic
    • Apply reference style on the content image without any distorting or hurting the fine structure details.

​

6 of 34

Problem Statement �

  • Given two images (content image and style image) and produces a new image that has the content of the first image and the style and colors of the second image.
  • Redrawing the content picture with given style, the main problem is to detect the similar objects so that the output picture looks like a real picture has been taken with a camera.

Content Style

7 of 34

Motivation ��

​

  • The rapid advance in Deep Learning
  • Contribute in the new emerging field of neural style transfer which is started by Gatys in 2016
  • It is attracted field in academia as it needs improving or extending NST algorithm.
  • It’s hard problem for graphic designers that needs advanced level to be able to redraw it so it has high cost and time .
  • Also in industry Like Prisma, Ostagram and Deep Forger applications [deleteable]

8 of 34

Previous Work

​

  • A Closed-form Solution to Photorealistic Image Stylization.
  • Deep Photo Style Transfer [deleteable]
  • WCT2
  • Semantic Segmentation
  • Mask R-CNN

​

9 of 34

WCT2

  • Jaejun et al proposed Wavelet Corrected Transfer (WCT2)
  • Based on whitening and coloring transforms.
  • very fast with good quality.
  • Avoids spatial distortion in image content
  • However :
    • It can be seen as if a filter is applied to the entire photo.

​

10 of 34

Image Segmentation�

  • Using convolutional neural networks (CNNs), a deep learning technique called semantic segmentation lets you associate every pixel of an image with a class label. Applications for semantic segmentation include autonomous driving, industrial inspection, medical imaging, and satellite image analysis.�

11 of 34

Segmentation

  • Partitioning a digital image into multiple segments (sets of pixels, also known as image objects).

​

  • The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze.

​

  • We are using image segmentation to apply style to an input image, we will need to map each object on the style image to a similar object in the content image .

�

​

12 of 34

Types of Image Segmentation

13 of 34

Older Versions Of CNN VS Faster CNN

  • Starting from R-CNN till Fast CNN they

used selective search and it's output was

a picture in box using selective search

  • Faster CNN replaced the selective

search with (Region Proposal Network )

RPN and the convNet with ResNet.

​

Starting from R-CNN till Fast CNN they used selective search and it's output was a picture in box using selective search which has to search in 2000 picture Faster CNN replaced the selective search with (Region Proposal Network ) RPN and the convNet with ResNet

�

Starting from R-CNN till Fast CNN they used selective search and it's output was a picture in box using selective search which has to search in 2000 picture Faster CNN replaced the selective search with (Region Proposal Network ) RPN and the convNet with ResNet

�

Starting from R-CNN till Fast CNN they used selective search and it's output was a picture in box using selective search which has to search in 2000 picture Faster CNN replaced the selective search with (Region Proposal Network ) RPN and the convNet with ResNet

�

14 of 34

Semantic Segmentation

  • Not just recognition , labeling pixels as well 
  • Eech pixel of image is labelled with class label 
  • Passing the pixel to CNN and output it’s class label 

�

15 of 34

Mask R-CNN

  • Proposed by FaceBook Ai Research in 2018
  • Improvement over Faster R-CNN that is widely used for object detection.
  • Can be used for instance and semantic segmentation.
  • Uses different backbone architecture.

​

16 of 34

The Proposed Model

17 of 34

Proposed Model Phase 1

​

  • Semantic Segmentation using Mask R-CNN .
  • With ResNet 101 and Feature Pyramid Network (FPN) as its backbone.
  • Pre Trained on Common Objects in Context (COCO) dataset

​

​

​

18 of 34

Coco DataSet

  • Large-scale dataset that is used in object detection and segmentation.
  • Contain images that have everyday object from everyday scene.
  • Size is approximately 20 GiB.
  • It defines 133 Panoptic annotation classes.

19 of 34

Semantic Segmentation using Mask R-CNN

Proposed Model Phase 1

with ResNet 101 and Feature Pyramid Network (FPN) as its backbone

20 of 34

Categorizing segmentation classes

Appling new classes category

​

21 of 34

New segmentation classes

22 of 34

Detecting Small Area

Identify small area that is under a given threshold

23 of 34

Predicting new classes for small area

​

Predicting new classes for small area

24 of 34

Applying Filter window

Filter Equation

25 of 34

26 of 34

27 of 34

Results and Discussion �

28 of 34

Results and Discussion �

29 of 34

Results and Discussion �

30 of 34

Results and Discussion �

31 of 34

Other Challenges

  • Open source code not maintainable
    • Different technologies versions (PyTorch, Tensorflow)

​

  • Acquiring computation power and GPU
    • Paid clouds services (Amazon Web Services [AWS], Google cloud, MS Azure)
    • Free clouds (Google Colab)

​

  • Storage
    • Dataset size (20 to 300 GB)
    • Google Drive using University premium account with Colab

​

​

32 of 34

Conclusion

33 of 34

Future Work�

  • Appling Photorealistic Style transfer to video.
    • Temporal Consistency in frames
  • Better Segmentation for Trees
  • Human body parts segmentation.
  • Scale up the styling to be inherited from multiple style images instead of one image.

​

​

​

34 of 34

Questions