A Friendly Introduction to Machine Learning with ml5
Joey Lee | ml5js.org/
All Materials can be found:
github.com/ml5js/ml5-friendly-intro-to-ml-2019f
→ Hello! (5 mins)
→ Super basic intro to Machine Learning (30 mins)
→ Intro to ml5.js (15 mins)
→ 😴 10 Min Break
→ Coding session (~45 mins)
→ Build a quick-and-dirty demo (~30 mins)
→ Present demo (20 mins)
→ Q&A, Closing notes + Farewell (10 mins)
Hello , my name is Joey Lee .
This is my 🌟first🌟 time at CMU.
The thing I’m most excited about machine learning is the potential to augment creativity & do social & environmental good.
The thing that worries me the most about machine learning is that innovation is outpacing governance.
🌈 Creative/Design technologist and geographer
💻 https://jk-lee.com
AI/ML in the world
We increasingly interact with machine learning systems every day.
Our lives -- e.g. how we access content, who gets access to services, and how we get from A to B -- are being shaped by the data we produce and the people using our data.
These increasingly algorithmically and data driven interactions reflect an assemblage of social and technical forces that almost always benefit some and harm others.
AI/ML in the world
Let’s look at a few examples of ML in context:
(we won’t talk the below yet)
Research | creating datasets
Research | creating new models
Research | creating new models
Research | creating new models
Research | applied ML
Research | ML auditing
“The importance of just looking at your data”
Environment
Environment | physical
Environment | physical
Environment | surveillance
Communities | IRL
Communities | IRL
Communities | IRL
Communities | online
Policies
Policies
Policies (lack of)
Education | explainers
Education | explainers
Education | (new) open courseware
Education | ML edu for youth
Education | ML edu for youth
Art/Design
Art/Design
Art/Design
Art/Design
AI/ML in the world
“...by not considering everything, we are forgetting something.”
- J Paul Neeley
AI/ML in sci-fi
AI/ML in reality
AI/ML in reality
Overview of terms and applications
What is Artificial intelligence?
Source: Jason's Machine Learning 101 - Link
Artificial Intelligence
AI is the science of making things “smart”:
The attempt to achieve (human) intelligence in machines
A broad term for getting computers to perform human tasks
Source: https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/
General AI v.s. Narrow AI
General AI: fabulous machines that have all our senses (maybe even more), all our reason, and think just like we do.
Most of AI today is Narrow AI systems that can only do one (or a few) defined things as well or better than humans.
Source: https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/
Machine Learning
One of the Approaches to Achieve Artificial Intelligence
A field of computer science that uses statistical techniques to give computer systems the ability to "learn" with data, without being explicitly programmed.
DATA, DATA, DATA, a lot of Data
Source: https://en.wikipedia.org/wiki/Machine_learning
Traditional Programming V.S. Machine Learning
Source: https://machinelearningmastery.com/basic-concepts-in-machine-learning/
Deep Learning
A technique for implementing Machine Learning
Deep learning -
Source: https://developer.nvidia.com/deep-learning
Deep Learning
Source: https://developer.nvidia.com/deep-learning
An example of Deep Learning
Source: http://www.deeplearningbook.org/contents/ml.html
Types of ML systems’ learning styles
Supervised Learning
Unsupervised Learning
Reinforcement Learning
Supervised Learning
Training data is labeled
Source: https://slideplayer.com/slide/12369271/
Unsupervised Learning
The machine learn from an unlabelled data set.
Unsupervised learning can be grouped into clustering and dimensionality reduction.
Source: https://slideplayer.com/slide/12369271/
Unsupervised Learning
The machine learn from an unlabelled data set.
Unsupervised learning can be grouped into clustering and dimensionality reduction.
Source: https://slideplayer.com/slide/12369271/
Reinforcement Learning
The machine learn by trial-and-error through reward or punishment.
Types of ML outputs
Classification: Predict a class/label (cat or dog?)
Regression: Predict continuous values (price of a stock)
Clustering: Most similar other examples (Amazon related products recommendation)
Sequence Prediction: Predict the next word/sentence
Use Cases
Creative Examples of ML - Drawing
Source: https://experiments.withgoogle.com/ai
Creative Examples of ML - Images
Creative Examples of ML - Mobile
Source: https://experiments.withgoogle.com/ai
Creative Examples of ML - Physical Computing
How are ML applications made?
Is there a good way to differentiate between cats, dogs, and badgers?
The machine learning process
Data collection
Data preparation
Training a machine learning model
Using the ML model in context
E.g. an artist writes a script to automatically download all images of cats, dogs, and badgers from the internet.
E.g. the artist uses a crowd tasking service to 1. confirm that the image is either a cat, dog or badger, 2. Tag the image with the animal type, 3. Crop the image of the animal to a square aspect ratio. The artist then resizes all the images to 64x64px
E.g. the artist: 1. Uses a popular machine learning framework called Tensorflow in python to create a neural network. To create a neural network the artist, writes code that specifies image processing and data crunching methods that aim to pull out unique features for each animal type in the given images. The artist then 2. Iteratively refines and tests the model to achieve the highest accuracy.
E.g. the artist opensources this new pre-trained dog/cat/badger model. This comes just in time since the ASPCA in NYC has been in great need of a method to differentiate between the animals in their shelters. The artist is hired to create a web application that uses this model to label their shelter animals “cat”, or “dog” or “badger”.
The machine learning process
Data collection
Data preparation
Training a machine learning model
Using the ML model in context
E.g. an artist writes a script to automatically download all images of cats, dogs, and badgers from the internet.
E.g. the artist uses a crowd tasking service to 1. confirm that the image is either a cat, dog or badger, 2. Tag the image with the animal type, 3. Crop the image of the animal to a square aspect ratio. The artist then resizes all the images to 64x64px
E.g. the artist: 1. Uses a popular machine learning framework called Tensorflow in python to create a neural network. To create a neural network the artist, writes code that specifies image processing and data crunching methods that aim to pull out unique features for each animal type in the given images. The artist then 2. Iteratively refines and tests the model to achieve the highest accuracy.
E.g. the artist opensources this new pre-trained dog/cat/badger model. This comes just in time since the ASPCA in NYC has been in great need of a method to differentiate between the animals in their shelters. The artist is hired to create a web application that uses this model to label their shelter animals “cat”, or “dog” or “badger”.
The machine learning process
Data collection
Data preparation
Training a machine learning model
Using the ML model in context
E.g. an artist writes a script to automatically download all images of cats, dogs, and badgers from the internet.
E.g. the artist uses a crowd tasking service to 1. confirm that the image is either a cat, dog or badger, 2. Tag the image with the animal type, 3. Crop the image of the animal to a square aspect ratio. The artist then resizes all the images to 64x64px
E.g. the artist: 1. Uses a popular machine learning framework called Tensorflow in python to create a neural network. To create a neural network the artist, writes code that specifies image processing and data crunching methods that aim to pull out unique features for each animal type in the given images. The artist then 2. Iteratively refines and tests the model to achieve the highest accuracy.
E.g. the artist opensources this new pre-trained dog/cat/badger model. This comes just in time since the ASPCA in NYC has been in great need of a method to differentiate between the animals in their shelters. The artist is hired to create a web application that uses this model to label their shelter animals “cat”, or “dog” or “badger”.
The machine learning process
Data collection
Data preparation
Training a machine learning model
Using the ML model in context
E.g. an artist writes a script to automatically download all images of cats, dogs, and badgers from the internet.
E.g. the artist uses a crowd tasking service to 1. confirm that the image is either a cat, dog or badger, 2. Tag the image with the animal type, 3. Crop the image of the animal to a square aspect ratio. The artist then resizes all the images to 64x64px
E.g. the artist: 1. Uses a popular machine learning framework called Tensorflow in python to create a neural network. To create a neural network the artist, writes code that specifies image processing and data crunching methods that aim to pull out unique features for each animal type in the given images. The artist then 2. Iteratively refines and tests the model to achieve the highest accuracy.
E.g. the artist opensources this new pre-trained dog/cat/badger model. This comes just in time since the ASPCA in NYC has been in great need of a method to differentiate between the animals in their shelters. The artist is hired to create a web application that uses this model to label their shelter animals “cat”, or “dog” or “badger”.
The machine learning process
Data collection
Data preparation
Training a machine learning model
Using the ML model in context
The machine learning process
Data collection
Data preparation
Training a machine learning model
Using the ML model in context
Quite likely, this is where you’ll be interacting with machine learning - as a user of pre-trained models. You’ll use these models together with software like ml5.js to build applications that can do things like image classification or generate images, and more!
It used to be much harder/impossible to do this stuff
But now the ecosystem of tools/edu gets better
APIs for ML
Friendlier APIs for ML
Increasing educational opportunities for understanding ML
GUI based ML tools starting to become more available
Cloud computing gets easier / cheaper
Getting started
Ok, sooo if I wanted to make an online unicorn racing game that allows players to use body gestures to control their players, how would I do that?
Friendly Machine Learning for the Web
A neighborly approach to creating and exploring artificial intelligence in the browser.
Developing ml5 is not just about developing machine learning software, it is about making machine learning approachable for a broad audience of artists, creative coders, and students. The library provides access to machine learning algorithms and models in the browser, building on top of TensorFlow.js with no other external dependencies.
ml5.js is heavily inspired by Processing and p5.js.
ml5js.org
github.com/ml5js
ml5 contributors
tensorflow.org/js
Tensorflow allows you to use your browser’s GPU to do calculations!
CPU
GPU
Images via Book of Shaders - https://thebookofshaders.com/01/
ml5.js in 3 friendly steps
Step 1: Call your ml5 function.
Step 2: Apply your ml5 function - e.g. to an image, video, or text.
Step 3: Do something with the results.
Step 1: Call your ml5 function.
Step 2: Apply your ml5 function - e.g. to an image, video, or text
Step 3: Do something with the results
ml5.js in 3 friendly steps
const myClassifier = ml5.imageClassifier(‘MobileNet’, modelReadyFunction);
Step 1: Call your ml5 function.
Step 2: Apply your ml5 function - e.g. to an image, video, or text
Step 3: Do something with the results
ml5.js in 3 friendly steps
const myClassifier = ml5.imageClassifier(‘MobileNet’, modelReadyFunction);
function modelReadyFunction(){
myClassifier.classify(myCatImage, gotResultsFunction);
}
ml5.js in 3 friendly steps
Step 1: Call your ml5 function.
Step 2: Apply your ml5 function - e.g. to an image, video, or text.
Step 3: Do something with the results.
const myClassifier = ml5.imageClassifier(‘MobileNet’, modelReadyFunction);
function modelReadyFunction(){
myClassifier.classify(myCatImage, gotResultsFunction);
}
function gotResultsFunction(error, results) {
// an array of objects with “label” and “confidence”
// [ { label: ‘cat’, confidence: 0.74 } ]
console.log(results);
}
We also support ES6 promises and async/await!
Step 1: Call your ml5 function.
Step 2: Apply your ml5 function - e.g. to an image, video, or text.
Step 3: Do something with the results.
const myClassifier = await ml5.imageClassifier(‘MobileNet’);
const results = await myClassifier.classify(myCatImage);
// An array of objects with “label” and “confidence”
// [ { label: ‘cat’, confidence: 0.74 } ]
console.log(results);
And preload() with p5.js!
let myClassifier;
let catImg;
function preload() {
myClassifier = ml5.imageClassifier(‘MobileNet’);
catImg = loadImage(‘cat.jpg’);
}
function setup() {
myClassifier.classify(catImg, gotResults);
}
ml5.js version 0.1.0
image
sound
text
helpers
imageClassifier
poseNet
styleTransfer
YOLO
pitchDetection
LSTM
word2vec
featureExtractor
ml5.js version 0.4.0
image
sound
text
helpers
imageClassifier🌈
poseNet🌈
styleTransfer
YOLO
BodyPix🦄
UNET🦄
Pix2Pix🦄
CVAE🦄
DCGAN🦄
SketchRNN🦄
FaceAPI 🦄
Pitch Detection
soundClassifier🦄
CharRNN🌈
Word Vectorization
Sentiment🦄
FeatureExtractor🌈
KNNClassifier🦄
NeuralNetwork🦄
Kmeans 🦄
🌈 - updates and new functionality
🦄 - new feature
imageClassifier(‘MobileNet’)
ported by Cristobal Valenzuela
const classifier = ml5.imageClassifier(‘MobileNet’);
classifier.classify(video, gotResult);
function gotResult(error, result) {
console.log(result);
}
imageClassifier(‘DoodleNet’)
created by Yining Shi
const classifier = ml5.imageClassifier(‘DoodleNet’);
classifier.classify(canvas, gotResult);
function gotResult(error, result) {
console.log(result);
}
PoseNet
ported by Cristobal Valenzuela, Maya Man, Dan Oved.
const posenet = ml5.poseNet(video);
posenet.on('pose', function(results) {
poses = results;
});
function draw() {
if (poses.length > 0) {
circle(poses[0].nose.x, poses[0].nose.y);
}
}
const uNet = ml5.uNet('face');
uNet.segment(video, gotResult);
function gotResult(error, result) {
image(result.image, 0, 0);
}
UNet
ported by Zaid Alyafeai
BodyPix
ported by Joey Lee & Dan Oved
const bodypix = ml5.bodyPix();
bodypix.segment(video, gotResult);
function gotResult(error, result) {
image(result.image, 0, 0);
}
Style Transfer
ported by Yining Shi
const cubist = ml5.styleTransfer(‘models/cubist’, modelReady);
function modelReady() {
cubist.transfer(video, gotImage)
}
function gotImage(error, result) {
image(result.image, 0, 0);
}
pix2pix
ported by Yining Shi
CVAE (conditional variational autoencoder)
ported by Wenhe Li & Derek (Dingsu) Wang
const cvae = ml5.CVAE('model/quick_draw/manifest.json');
cvae.generate(‘alarm clock’, gotImage);
function gotImage(error, results) {
image(results.image, 0, 0);
}
DCGAN (deep convolutional generative adversarial network)
ported by YG Zhang, Rui An & Joey Lee
const dcgan = ml5.DCGAN('model/geo/manifest.json');
dcgan.generate(gotImage);
function gotImage(error, results) {
image(results.image, 0, 0);
}
SketchRNN
ported by Reiichiro Nakano and Dan Shiffman
const model = ml5.sketchRNN('cat');
model.generate(gotStroke);
function gotStroke(error, path) {
if (pen == ‘down’) {
line(x, y, x + path.dx, y + path.dy);
}
x += path.dx;
y += path.dy;
pen = path.pen;
}
YOLO
ported by Cristobal Valenzuela
const yolo = ml5.yolo();
yolo.detect(video, gotResults);
function gotResults(error, results) {
for (let item of results) {
rect(item.x, item.y, item.w, item.h)
}
}
SoundClassifier(‘SpeechCommands18w’)
ported by Yining Shi
let classifier = ml5.soundClassifier(‘SpeechCommands18w’);
classifier.classify(gotResult);
function gotResult(error, result) {
labelDiv.html(result[0].label);
confidenceDiv.html(result[0].confidence);
}
PitchDetection
ported by Hannah Davis
const audioContext = getAudioContext();
const mic = new p5.AudioIn();
const pitch = ml5.pitchDetection('crepe/model.json', audioContext , mic);
pitch.getPitch(gotPitch);
function gotPitch(error, results) {
console.log(results.frequency);
});
CharRNN
ported by Cristobal Valenzuela and Memo Atken
Sentiment(‘MovieReviews’)
ported by Itay Niv
const sentiment = ml5.sentiment('MovieReviews');
const prediction = sentiment.predict('I am so happy!');
sentimentResult.html('Sentiment score: ' + prediction.score);
Word2Vec
ported by Cristobal Valenzuela, Dan Shiffman, and Jenna Xu
Transfer Learning
KNNClassifier¹ & FeatureExtractor² + Teachable Machine
Yining Shi¹ & Cristobal Valenzuela²
Face-Api (coming soon)
ported by Joey Lee
const faceapi = ml5.faceApi();
faceapi.detect(video, gotResults);
function gotResults(error, results) {
drawLandmarks(results);
}
DIY Neural Network
Wrapped by Dan Shiffman, Yining Shi, & Joey Lee
const options = {
dataUrl: ‘data/co2_emissions.csv’,
input:[‘population’],
output: [‘co2_emissions’],
task: ‘regression’
};
// Create Neural Network
neuralNetwork = ml5.neuralNetwork(options);
neuralNetwork.normalizeData();
neuralNetwork.train(options, whileTraining, doneTraining);
neuralNetwork.predict([10000], gotResults);
18 releases
50+ contributors
80+ watching
2600+ stars
250+ pull requests
130+ “used by” / 200+ forks
Apply this to games, tools, and more!
https://wp.nyu.edu/lillianritchie/2019/05/13/nature-of-code-final-food-chain-game/
“Technology is not enough.”
“Consider the technology as a tool which, in itself, could do nothing.”
“Treat the technology as something that everyone on the team could learn, understand, and explore freely.”
— Red Burns, founder of ITP, 1925-2013
p5js.org
ml5 x edu
ml5.js is being used as an entry point for students from highschool to grad school to un-school to give people a chance to use state-of-the-art machine learning algorithms and discuss implications of ai/ml in society.
Ml5.js community: https://ml5js.org/community
ml5.js plays nicely with other web ML tools
g.co/teachablemachine19
runwayml.com
Artificial Intelligence for Augmented Creativity
Runway is a toolkit that allows creators of all kinds to use artificial intelligence in an intuitive way.
RainbowGAN
Trained on 5,000 images with the “rainbow” keyword, sorted by relevance from the Flickr API. https://github.com/antiboredom/flickr-scrape/
StyleGAN: A Style-Based Generator Architecture for Generative Adversarial Networks Tero Karras (NVIDIA), Samuli Laine (NVIDIA), Timo Aila (NVIDIA)
https://github.com/NVlabs/stylegan
Model trained by Anastasis Germanidis.
ml5.js welcomes all contributions!
10 mins break!
Code-along / DIY ML
Example 1: image classifier
Example 2: PoseNet
Example 3: FaceApi
Example 4: Neural Network - json data / classification
Example 5: Neural Network - real-time / regression
You have 30 min to make a thing!
projects showcase
All Materials can be found:
github.com/ml5js/ml5-friendly-intro-to-ml-2019f