1 of 14

Multi-Instance Theme Rendering�with neural network

Richard, Yizhou, and Zhaoyi

2 of 14

Neural Style Transfer

  • Takes two images as input: content, style
  • Uses pre-trained CNN to transfer styles from a given image to another
  • Outputs an image such that it looks like the content image, but “painted” in the style of the style image

​

3 of 14

Motivation

  • Seek more practical uses of NST
  • Only render desirable objects (e.g. people)
  • Render different objects with different styles
  • Preserve more original details of the objects

4 of 14

Our approach

  • Objective:
  • For a given input image with k instances, we want to generate an output image where k instances are applied with different theme filters individually selected by the user.
  • Components:
  • Theme rendering + Instance segmentation

5 of 14

Theory base

  • Models
    • Pastiche model: The model to be trained on a content image dataset and style images.
    • A pre-trained neural network: Pre-trained on ImageNet. Used here for feature extraction.

​

  • Keys:
    • Conditional instance normalization
    • loss functions

​

Total = λs*Ls(p) + λc*Lc(p)

Our theme rendering approach is partially based on A Learned Representation For Artistic Style by Dumoulin et al (2017).

​

6 of 14

Open questions remained:

  • Pastiche model Architecture
  • Ways to perform conditional instance normalization
  • Model selection for feature extraction
  • Layer selections for content and style loss computation
  • Balancing weights λs and λc

7 of 14

Theme rendering�

Feature extraction

8 of 14

Theme rendering

Loss computation

 

9 of 14

 

 

(Fix loss weights and hyper-parameters)

+

10 of 14

Theme rendering

Conditional instance normalization with random cell expansion

“beehive”

Advantages:

Training: Reduce training time in a large margin.

​

Inferencing: Input one image, and output multiple theme rendered images with different selected filters all together with one forward pass.

​

“jungle”

11 of 14

Instance segmentation �

Training time random theme blending

For each ROI with coordinate (x, y), a filter i is applied (i <- (x, y)) to compute patch-wise loss respectively and added together.

​

However, unstable results (ROI for each training image is different). Slow to train.

​

​

12 of 14

Instance segmentation �

Inference time instance segmentation

​

Mask-RCNN

1

3

2

4

5

13 of 14

Results

+

ε = 2.6

​

14 of 14

Review

  • New Pastiche model for our theme rendering
  • CIN with Random cell expansion
  • Light weight feature extractor
  • Different style loss and content loss computation schemes
  • R-CNN for instance segmentation
  • Introduced ε to tweak filter levels (act as a slide bar)

​

​

​

​

​

​

https://github.com/cv639-project