1 of 1

Artists invest a lot of time and money into testing new ideas for works of art

Barrier for artists whose main source of income is their art
Increases the price of artwork

Objective: create a neural network that converts a photograph from to an artistic style.

ex. a photograph of a face is modified to be in the impressionist style

Makes it easier for artists to test ideas for color palettes and textures

See a model of what the final product would look like
Reduce the cost of drafting.

Problem Statement

Engineering Goal

Two datasets of around 200 images each were obtained

Portrait photographs of people in color
Portrait sketches

A Cycle Generative Adversarial Network (CycleGAN) was built

A GAN has two components, one that attempts to create something similar the data fed in (generator) and one that tries to distinguish between training data and generated data (discriminator).

The networks exchange outputs and try to “beat” each other
Generator that creates data similar to training data
Discriminator that distinguishes between real and fake (generated) data

A CycleGAN uses two generators and two discriminators to convert one type of data into another type of data and vice versa.

Generators converts data from one category into the other category (first to second or second to first)
Discriminators differentiate between converted and original data from one category
Uses cycle consistency, measured by cyclical loss

Image generated by one GAN is put into the other GAN, and the result is compared to the original image
A better CycleGAN means result will similar to the original image

CycleGAN takes in the datasets and trains the generators and discriminators on each dataset for 50 epochs

Each pair of models creates a composite model that updates the generator

Eventually able to modify images so that they match the “opposite” dataset - photograph of someone “stylized” into a black and white sketch
Success measured by the accuracy of the discriminators

Ideal value is 50% - discriminator has a random chance of guessing whether a given image is real or fake
Suggest that the generator is stronger than the discriminator; it can “trick” the discriminator or basically is good at making fakes
Goal is to have a stronger generator and a weaker discriminator since the network’s major purpose is to create artwork.

Building the Network

While the results shown in Figure 1 produced clear and recognizable images of the people in the original color photo, the results in Figure 2 were mostly incoherent. This is likely due to the possibility of overtraining, since the composition of the style and content datasets did not fully match: the style datasets had an assortment of sketches from portraits and full-body drawings to a few still life observations. As a result, the network may have had less relevant “examples” to learn from.
The results after 23 epochs for the black-and-white to color translation were far more recognizable, further confirming the possibility of overtraining.

Problem was resolved by including a more specific style dataset consisting of only portrait sketches.
Better results were obtained due to the similarity in subject and composition in both the style and content datasets, which enabled the model to focus on conversion between different colors and textures instead of different types of compositions.
Some uncontrolled events such the outpacing of the generator by the discriminator did occur. This likely happened because it was very easy for the discriminator to differentiate the generated images and the original images, due to the colors.

This may be solved by lowering the weights, making the discriminator would train slower. This would result in clearer generations since the generator is able to develop more confidence and train faster, as opposed to constantly being outsmarted by the discriminator.

Modifying the Network

Results and Analysis

The results of the model are as expected: the colors of the photographs are converted to grayscale and some textures were added.

Major features and light values were preserved

Model used red and brown to replace the medium gray tones when converting to color

Unable to reproduce images in a full color scheme, but similar to pastel or brown charcoal sketches
Useful for artists: highlights light and shadow in a composition

Translations of some nonhuman features (ex. flowers) were unclear

because rare in the training dataset
Clothing folds managed well - appeared often in data

Conclusions

The design goals for this research were largely met. The model transformed color photographs into a sketch-like style, although it wasn’t perfect.

Measured by qualitative judgment

The original objective to transform photographs into more sophisticated art styles was unmet.
Translation to sketch style provides a proof of concept

More sophisticated transformations are possible!

Provides improvements over current translations methods like DeepDream and Neural Style Transfer.

DeepDream magnifies patterns in an image
Neural Style Transfer combines two images, but drops texture
This model applies style to an image - texture and color schemes kept

Training the model with significantly larger datasets, will create a tool for artists to visualize a given work in different artistic styles.
Currently, the model can translate color photographs into black and white sketches with artifacts such as varying line marks which mimic different pencil sketching techniques and crosshatching.
Model can be implemented into a user interface

Accepts an inputted photograph and converts it into another style
Save time and money - don’t have to draft by hand

Future Extensions

Training datasets with new artistic styles will be fed into the CycleGAN in order to translate images into different styles such as Baroque or Impressionism.
If the CycleGAN is able to complete the above conversions with a high success rate, a Progressive GAN will be trained for generating original artwork, instead of converting images, by feeding in many paintings of portraits in similar styles.
A Progressive GAN will enable the model to start with a low quality image and gradually improve the quality of the image, better picking up different compositions and textures.

Final goal: network learns a style and then applies it to an image
Based on reviewed research articles, a GAN will produce optimal results.
To work towards this goal, the first attempt was to stylize an image by combining it with another image, using Neural Style Transfer. Then a regular GAN was created.
This was modified into a more specific CycleGAN
The next focus was learning a style.

An uncomplicated style was used: black and white sketches
Dataset with black and white style images and similar subject color images was used to train the model, so that one style of image can be converted to the other.

More complicated styles could be attempted upon success, such as baroque and Impressionism.

Olivia Li and Shreya Paliwal

Stylizing Images Using Neural Networks

Resources

Sketch or black and white style

Portrait of a Man (Konrad Peutinger) by Albrecht Durer

Laptop with increased GPU capabilities
GPU: NVIDIA GeForce RTX 2080 Max-Q Design
Software packages: tensorflow, numpy, keras, matplotlib
Coding platform: Google Colab, Spyder
Datasets: Kaggle, Pinterest, MNIST data imported from keras

Impressionist style

Portrait of the Actress Jeanne Samary by Pierre-Auguste Renoir

An NST was created to combine two images into one, by applying the style of one image onto the content of another.

The “style” image provides the color palette and texture while the content image provides the composition.

Next, a standard GAN was created.

Predicted function values between range values of -1 and 1 after being given training data of about 100 values for a function in this range.

This GAN was modified to accept image inputs, for which the MNIST data set of handwritten digits was used.

Procedures

Diagram of how a CycleGAN functions:

Loss function graphs which map out the progress of the discriminators and generators:

www.PosterPresentations.com

(—THIS SIDEBAR DOES NOT PRINT—)

DESIGN GUIDE

This PowerPoint 2007 template produces a 36”x48” trifold presentation poster. You can use it to create your research poster and save valuable time placing titles, subtitles, text, and graphics.

We provide a series of online tutorials that will guide you through the poster design process and answer your poster production questions. To view our template tutorials, go online to PosterPresentations.com and click on HELP DESK.

When you are ready to print your poster, go online to PosterPresentations.com�

Need assistance? Call us at 1.510.649.3001

QUICK START

Zoom in and out

As you work on your poster zoom in and out to the level that is more comfortable to you.

Go to VIEW > ZOOM.

Title, Authors, and Affiliations

Start designing your poster by adding the title, the names of the authors, and the affiliated institutions. You can type or paste text into the provided boxes. The template will automatically adjust the size of your text to fit the title box. You can manually override this feature and change the size of your text.

TIP: The font size of your title should be bigger than your name(s) and institution name(s).

�

Adding Logos / Seals

Most often, logos are added on each side of the title. You can insert a logo by dragging and dropping it from your desktop, copy and paste or by going to INSERT > PICTURES. Logos taken from web sites are likely to be low quality when printed. Zoom it at 100% to see what the logo will look like on the final poster and make any necessary adjustments.

TIP: See if your school’s logo is available on our free poster templates page.

Photographs / Graphics

You can add images by dragging and dropping from your desktop, copy and paste, or by going to INSERT > PICTURES. Resize images proportionally by holding down the SHIFT key and dragging one of the corner handles. For a professional-looking poster, do not distort your images by enlarging them disproportionally.

Image Quality Check

Zoom in and look at your images at 100% magnification. If they look good they will print well.

ORIGINAL

DISTORTED

Corner handles

Good printing quality

Bad printing quality

QUICK START (cont.)

How to change the template color theme

You can easily change the color theme of your poster by going to the DESIGN menu, click on COLORS, and choose the color theme of your choice. You can also create your own color theme.

You can also manually change the color of your background by going to VIEW > SLIDE MASTER. After you finish working on the master be sure to go to VIEW > NORMAL to continue working on your poster.

How to add Text

The template comes with a number of pre-formatted placeholders for headers and text blocks. You can add more blocks by copying and pasting the existing ones or by adding a text box from the HOME menu.

Text size

Adjust the size of your text based on how much content you have to present. The default template text offers a good starting point. Follow the conference requirements.

How to add Tables

To add a table from scratch go to the INSERT menu and �click on TABLE. A drop-down box will help you select rows and columns.

You can also copy and a paste a table from Word or another PowerPoint document. A pasted table may need to be re-formatted by RIGHT-CLICK > FORMAT SHAPE, TEXT BOX, Margins.

Graphs / Charts

You can simply copy and paste charts and graphs from Excel or Word. Some reformatting may be required depending on how the original document has been created.

How to change the column configuration

RIGHT-CLICK on the poster background and select LAYOUT to see the column options available for this template. The poster columns can also be customized on the Master. VIEW > MASTER.

How to remove the info bars

If you are working in PowerPoint for Windows and have finished your poster, save as PDF and the bars will not be included. You can also delete them by going to VIEW > MASTER. On the Mac adjust the Page-Setup to match the Page-Setup in PowerPoint before you create a PDF. You can also delete them from the Slide Master.

Save your work

Save your template as a PowerPoint document. For printing, save as PowerPoint or “Print-quality” PDF.

Print your poster

When you are ready to have your poster printed go online to PosterPresentations.com and click on the “Order Your Poster” button. Choose the poster type the best suits your needs and submit your order. If you submit a PowerPoint document you will be receiving a PDF proof for your approval prior to printing. If your order is placed and paid for before noon, Pacific, Monday through Friday, your order will ship out that same day. Next day, Second day, Third day, and Free Ground services are offered. Go to PosterPresentations.com for more information.

Student discounts are available on our Facebook page.�Go to PosterPresentations.com and click on the FB icon.

Berkeley CA 94710�posterpresenter@gmail.com