RunwayML & Google Colab
Derrick Schultz
Lia Coleman
SCHEDULE
WEEK 1 Intro to RunwayML
WEEK 2 Intro to Colab
WEEK 3 Types of GAN
WEEK 4 Creating Datasets
WEEK 5 (no class)
WEEK 6 Training StyleTransfer & NFP Models
WEEK 7 Training StyleGAN, MUNIT Models
WEEK 8 (no class)
WEEK 9 Inference
WEEK 10 Integrations, Additional Resources
TBD Show N Tell
TODAY: Training NFP & Style Transfer Models
BREAKOUT How did dataset making go?
LECTURE Training in Colab
DEMO Training NFP
DEMO Training StyleTransfer
HOMEWORK
BREAKOUT 10 MINS
Homework discussion:
How did dataset making go?
TRAINING ON COLAB
GOOGLE DRIVE + TRAINING
Some trainings can be done in < 10 hours, but often we’re talking days or weeks.
The problem with training sessions that last longer than your Colab period is that you can lose your files at any moment.
USUALLY...
COLAB SERVER
Hardware (GPU/CPU)
Dependencies (pip)
Git Repo
Files
GOOGLE DRIVE + TRAINING
Another problem you likely run into without Colab Pro is hard disk space on your Colab Server. (w/o Pro you get ~50GB.)
NFP can produce ~750MB per epoch
USUALLY...
COLAB SERVER
Hardware (GPU/CPU)
Dependencies (pip)
Git Repo
Files
All on server HD
GOOGLE DRIVE + TRAINING
Saving files directly to Google Drive means as soon as the file is written its saved in a location you won’t lose when your server disconnects.
NOW
COLAB SERVER
Hardware (GPU/CPU)
Dependencies (pip)
DRIVE
Git Repo
Files
GOOGLE DRIVE + TRAINING
Google Drive is virtual storage for Colab. You write to it without it counting against your Colab HD, but it’s accessible like* your Colab HD.
NOW
COLAB SERVER
Hardware (GPU/CPU)
Dependencies (pip)
DRIVE
Git Repo
Files
<—————————Disk storage split—————————>
NEXT FRAME PREDICTION
DATASET
TRAINING
TRAINING
TRAINING
And on until all pairs have been trained...
TRAINING
INFERENCE/TESTING/GENERATING
Predicted Frame
INFERENCE/TESTING/GENERATING
Predicted Frame
Predicted Frame
INFERENCE/TESTING/GENERATING
Predicted Frame
Predicted Frame
Predicted Frame
STEP ONE
Upload your video to Colab. I recommend the video be 1280x720.
The video’s dimensions must be multiples of 32px (sometimes it works if not but its safer if it is)
STEP TWO: SEPARATE FRAMES
Next we need to break the video into individual frames and create paired folders of images
STEP THREE: TRAIN MODEL
Training a model runs the model thru the GAN. Every time it runs thru the entire length of the video is called an epoch. By default the model trains for 200 epochs, but you may need to train it longer.
STEP FOUR: CONTINUE TRAINING
After ~20hours (on Colab Pro) your machine will disconnect. Reconnect your machine and run the first couple of cells. Skip over the frame extraction cell and run the cell that include --continue_train
STYLE TRANSFER
STYLE TRANSFER
Some style transfer models are trained (“Fast Style Transfer”) so that they work better in real time. My personal preference is for style transfers that don’t use scaling, ad instead use iterative and scaling processes.
Style
✕�Content
At its most basic:
python neural_style.py --content_img floracon1.jpg --style_imgs rodina.jpg
(any parameters not altered in a command will use defaults. In this case that means the max_size is 512, and iterations is 1000)
python neural_style.py --content_img floracon1.jpg --style_imgs rodina.jpg --max_size 1400 --max_iterations 500 --img_output_dir ./floracon1-rodina
python neural_style.py
--content_img floracon1.jpg
--style_imgs rodina.jpg
--max_size 1400
--max_iterations 500
--img_output_dir ./floracon1-rodina
--max_iterations 100
--max_iterations 200
--max_iterations 300
--max_iterations 400
--max_iterations 500
--max_iterations 600
--max_iterations 700
--max_iterations 800
--max_iterations 800
--style_scale 1.0
(default value)
--style_scale 0.75
--style_scale 0.5
--style_scale 0.25
--content_weight 0e0
--init_img_type random
(default: --seed 0
--style_scale 0.25)
--style_imgs davy2.jpg rodina.jpg
--style_imgs_weight 0.5 0.5
--style_imgs davy2.jpg,rodina.jpg
--style_imgs_weight 0.25 0.75
--style_imgs davy2.jpg,rodina.jpg
--style_imgs_weight 0.75 0.25
--original_colors
HOMEWORK
Have (Pix2Pix OR CycleGAN/MUNIT) OR StyleGAN �model for the following Monday
Pix2Pix:
CycleGAN/MUNIT:
StyleGAN:
HOMEWORK
Train an NFP model