Generating Images & Videos with ML
Start at 9:15pm
Lia Coleman
AI Artathon 2021 - Framed AI Art
Art, AI, education.
LIA COLEMAN
I do / have done work for:
Rhode Island School of Design (RISD)
RunwayML
NeurIPS Workshop on Creativity & Design
Partnership on AI
ML effects for Polae, a 2021 Tribeca Film Fest official selection
School for Poetic Computation, Babycastles
MIT
THIS IS AN
INTERACTIVE CLASS.
WHAT I NEED FROM YOU:
We all come from different levels-- and that’s awesome!
TODAY
Poll: Familiarity with Code, ML?
Quick Intro & Inspiration
INTRO TO FRAMED AI ART
ML BASICS & INSPIRATION
AI ART
Art that is made
using machine learning / AI
images, video, music, poetry, performance
The dataset
Video as �dataset
PROJECT INSPIRATION
FILM
DESIGN�FASHION
BOOKS
DESIGN
MACHINE LEARNING 101
GENERAL A.I.
Robots, Supercomputers,�Fiction.
NARROW A.I.
Code that does one thing really well.
ML PROCESS
DATASETS
TRAINING
TESTING
DATASETS
DATASETS
DATASETS
TRAINING
TRAINING
TRAINING
TRAINING
TESTING
TESTING
TESTING
Questions?
RUNWAY:
•“Photoshop for ML”
•GUI, no code
•Removes huge hurdles
•Interface improves functionality
RUNWAY ML
DATASETS
TRAINING
TESTING
RUNWAY:
•Pre-trained models
•Not every model
•Limited training
•Expensive to use
RUNWAY: ($ = USD)
•$.05 per min. testing
•$.005 per step*
•*training requires subscription ($15/m, $144/year)
Demo: �Exploring Runway
GROUP WORK
Explore the pre-trained models in Runway
Find at least 3 models to explore.
Write down:
Copy a slide below & show us what you made!
Share Your Result Here!
Sabrina Kaune
eddy
Image analysis
eddy
Image analysis
Picatso
Kevin
Share Your Result Here!
lia
Share Your Result Here!
Mohammed Alali
Share Your Result Here!
YOUR NAME!
BREAK
2) Google Colab
with StyleGAN
What is Colab?
Colab Pro*
• $10/month
• need a billing address from this list of countries
• 20 hours (vs. 10)
• Higher chance of � “good” GPU (P100)�
Colab vs RunwayML
WHAT IS A GAN?
Generative
Adversarial
Network
GENERATOR
DISCRIMINATOR
GENERATOR
DISCRIMINATOR
HERE’S A REAL �IMAGE, I SWEAR
YEAH, THAT’S NOT A REAL IMAGE
GENERATOR
DISCRIMINATOR
OK HERE’S A �REAL IMAGE
WAIT,MAYBE THIS ONE IS A REAL IMAGE?
History of GANs
What is a Latent Space?
LATENT SPACE
LATENT SPACE
LATENT SPACE
LATENT SPACE
LATENT SPACE
LATENT SPACE
StyleGAN uses
512 dimensions
(that’s like three dimensions, but way more)
INFERENCE
NOISE LOOP INTERPOLATION
GOOGLE COLAB + STYLEGAN
NOISE LOOPS
15 mins
groups
Beetle boys
eddy
Share Your Result Here!
https://drive.google.com/drive/my-drive
YOUR NAME(S)!
Share Your Result Here!
https://drive.google.com/drive/my-drive
YOUR NAME(S)!
Share Your Result Here!
https://drive.google.com/drive/my-drive
YOUR NAME(S)!
Share Your Result Here!
https://drive.google.com/drive/my-drive
YOUR NAME(S)!
What did you notice?
What was surprising or difficult?
Questions?
-> STYLEGAN MASTER COLAB NB <-
* = easy!
Where to find pretrained styleGAN models? Justin Pinkney’s Awesome Pretrained StyleGAN2
NOISE LOOP INTERPOLATION
LINEAR INTERPOLATION
LINEAR INTERPOLATION
LINEAR INTERPOLATION
FLESH DIGRESSIONS
PROJECTION
EXAMPLE: Projecting my face into the FFHQ faces pretrained model
PROJECTING IN�LATENT SPACE
PROJECT INTO�LATENT SPACE
Share Your Result Here!
YOUR NAME!
(TIME PERMITTING)
GROUP WORK
-> STYLEGAN MASTER COLAB NB <-
* = easy!
Share Your Result Here!
YOUR NAME!
Share Your Result Here!
YOUR NAME!
Share Your Result Here!
YOUR NAME!
3) Google Colab
with VQGAN + CLIP
VQGAN + CLIP
“demons are the powers and principalities of the air”
Ryan Murdock @advadnoun
“mind on fire”
“overshadowed”
VQGAN + CLIP
VQGAN’s latent space
VQGAN + CLIP
VQGAN’s latent space
VQGAN + CLIP
VQGAN’s latent space
VQGAN + CLIP IN GOOGLE COLAB
For step-by-step instructions, read this guide by @images_ai
“This was the text prompt”
YOUR NAME(S)!
“This was the text prompt”
YOUR NAME(S)!
“This was the text prompt”
YOUR NAME(S)!
What did you notice?
What was surprising or difficult?
Questions?
HAPPY AI-ART MAKING!
NEXT CLASS
Make a linear interpolation latent walk video.
Upload the video to the class shared Drive,
then share it in a slide.
15 mins
groups
LINEAR INTERPOLATION
LINEAR INTERPOLATION
HOW TO TRAIN YOUR OWN STYLEGAN MODEL
Lecture
DATASETS
Lecture
Why Datasets?
Datasets are the creativity in Machine Learning Art.
Datasets are the hardest part in Machine Learning Art.
Why Datasets?
What data do you uniquely have access to?
What skill sets do you have?
Pre-trained models are limiting
Pre-trained models only cover a small set of use cases.
You’re now a data scientist and ML researcher. Congrats :)
DATASET DIVERSITY
StyleGAN
What makes a good dataset?
What makes a good dataset?
WAYS TO MAKE DATASETS
B. Scrape existing media.
C. Use existing ready-made datasets.
B. Scraping existing media
Least-risky: Use media that is in the creative commons or public domain.
Ex: NASA*, Biodiversity Heritage Library
B. Scraping existing media
= using someone else’s data
“To work with generative systems is to be a curator...to curate a corpus is to value the contributions inside of it...they are still artworks individually, and the people who make them are still artists.
Curating your own corpora is to be able to deal with the original creators of your corpora as humans and as collaborators, not as datapoints.”
Everest Pipkin
Say thank you! :)
Everest Pipkin, i've never picked a protected flower (concrete unicode poems)
B. Scraping existing media
= using someone else’s data
C. Using an existing dataset
Think about:
Example: MegaFace Dataset
photo albums w/o permission
for FaceRec tech
BEST PRACTICES
HOMEWORK
Think about what your dataset could be. Post in Slack with your idea! Some approaches:
for next class.
DATASET COLLECTION DETAILS:
THINK: What data can you find in large enough quantity that is interesting to you?
FUN MEDIA
AI makers to follow
Artists
Curator & AI art organizer
ML music
Stay up to date- AI Artwork mailing list
BREAKOUT
Generate Images & Video from a StyleGAN model
BREAKOUT: GENERATE IMAGES & VIDEO FROM A STYLEGAN MODEL.
Share Your Result Here!
https://drive.google.com/drive/my-drive
YOUR NAME(S)!
Share Your Result Here!
https://drive.google.com/drive/my-drive
YOUR NAME(S)!
Share Your Result Here!
https://drive.google.com/drive/my-drive
YOUR NAME(S)!
Share Your Result Here!
https://drive.google.com/drive/my-drive
YOUR NAME(S)!
Image Segmentation
Image Segmentation
Image Segmentation
Image Segmentation
Household objects
natural landscapes
HANDS-ON:
Play with SPADE!
15 mins
Share Your Result Here!
YOUR NAMES!
Share Your Result Here!
YOUR NAMES!
Share Your Result Here!
YOUR NAMES!
�Vector Input with StyleGAN
Latent Space
Latent Space
Latent Space
Latent Space
Latent Space
Latent Space
Latent Space
StyleGAN uses
512 dimensions
(that’s like three dimensions, but way more)
Latent Space
Runway tries to visualize this �high-dimensional
space with a 2D image grid using the vector input.
LATENT SPACE
StyleGAN uses
512 dimensions
(that’s like three dimensions, but way more)
IN GROUPS:
Generate images with a StyleGAN model.
15 mins
groups
Share Your Results Here!
YOUR NAMES!
Share Your Results Here!
YOUR NAMES!
Share Your Results Here!
YOUR NAMES!
Share Your Results Here!
YOUR NAMES!
Share Your Results Here!
YOUR NAMES!
INTERPOLATION/
LATENT WALKS
LINEAR INTERPOLATION
LINEAR INTERPOLATION
LINEAR INTERPOLATION
IN GROUPS:
Make a latent walk video.
Upload the video to the class shared Drive,
then share it in a slide.
15 mins
groups
Share Your Result Here!
YOUR NAMES!
Share Your Result Here!
YOUR NAMES!
Share Your Result Here!
YOUR NAMES!
Share Your Result Here!
YOUR NAMES!
Share Your Result Here!
YOUR NAMES!
Share Your Result Here!
YOUR NAMES!
EXPLORE RUNWAY
20 mins
groups
BREAK!
Hold up! Why am I even making this?
Be honest with yourself about your goals for using AI, even if you're just looking to learn and play!
Checkpoint 1: Dataset
Where does the training data come from?
How diverse is the dataset?
Am I respecting data creators and subjects?
DISCUSSION
5 mins
groups
Checkpoint 2: Model Code
Whose code are you depending on for your work?
Am I respecting the people who contributed to the model code?
DISCUSSION
5 mins
groups
Checkpoint 3: Training Resources
What are the environmental costs of my training?
�
Training a single AI model can emit as much carbon as five cars in their lifetimes �(MIT Technology Review)
From Memo Atken’s The Unreasonable Ecological Cost of #CryptoArt�Let’s Talk Energy Usage of Generative Machine Learning (Derrick Schultz)�
�Let’s Talk Energy Usage of Generative Machine Learning (Derrick Schultz)�
Checkpoint 4: Publishing
Who might benefit from this work?
What are unintended consequences of releasing my model/code/dataset?
�How might I make my work accessible to others?
DISCUSSION
5 mins
groups
Best Practices
In Runway, other models to check out:
Automatic Sketch Colorization
Rotoscoping Green Screen
FUN MEDIA
10 mins
Breakout rooms
15 mins
groups