MACHINE LEARNING
FOR FRONT-END DEVELOPERS
Charlie Gerard / @devdevcharlie
Charlie Gerard
Front-end developer, Google Dev Expert & Mozilla Tech Speaker
WHAT IS MACHINE LEARNING
1
Giving the ability to computers to find patterns in data without being explicitly programmed
DIFFERENCE FROM TRADITIONAL PROGRAMMING
Example: Recognising spam emails
if(email.includes('viagra')){
filterAsSpam(email);
} else if(email.includes('v!agra')){
filterAsSpam(email);
} else if(...){
....
}
DIFFERENCE FROM TRADITIONAL PROGRAMMING
Example: Recognising spam emails
if(email.includes('viagra')){
filterAsSpam(email);
} else if(email.includes('v!agra')){
filterAsSpam(email);
} else if(...){
....
}
import * as spamModel from "model.json"
const model = load(spamModel);
const newEmail = getEmail();
const prediction = model.predict(newEmail);
prediction === 0 ? spam : not spam
ALGORITHMS
2
EXAMPLES OF ALGORITHMS
Supervised learning
Unsupervised learning
TYPES OF MACHINE LEARNING ALGORITHMS
Reinforcement learning
Supervised learning
TYPES OF MACHINE LEARNING ALGORITHMS
Reinforcement learning
Unsupervised learning
SUPERVISED LEARNING
Create predictive model based on a set of features
and labels.
SUPERVISED LEARNING
Create predictive model based on a set of features
and labels.
SUPERVISED LEARNING
Create predictive model based on a set of features
and labels.
SUPERVISED LEARNING
Create predictive model based on a set of features and labels.
SUPERVISED LEARNING
Create predictive model based on a set of features and labels.
Example: Predicting the price of a house
SUPERVISED LEARNING
Create predictive model based on a set of features and labels.
How you would classify each entry.
Example: Predicting the price of a house
Price
SUPERVISED LEARNING
Create predictive model based on a set of features and labels.
How you would classify each entry.
Example: Predicting the price of a house
Price
Characteristics of the entries in your dataset.
Number of rooms, floors, location, type of property, ...
SUPERVISED LEARNING
Create predictive model based on a set of features and labels.
How you would classify each entry.
Example: Predicting the price of a house
Price
Characteristics of the entries in your dataset.
Number of rooms, floors, location, type of property, ...
Mathematical representation of the outcome of your training.
Function that takes features as parameters and is able to predict a label as output.
UNSUPERVISED LEARNING
The ability to create predictions based only on a set of features.
UNSUPERVISED LEARNING
The ability to create predictions based only on a set of features.
UNSUPERVISED LEARNING
The ability to create predictions based only on a set of features.
Example: Predicting customer behaviour
APPLICATIONS
3
GENERATING ALT TEXT
FILTERING UNWANTED CONTENT
ACCESSIBILITY
MACHINE LEARNING
IN JAVASCRIPT
4
WHY
WHY
WHY
WHAT CAN YOU DO
Import an existing, pre-trained model.
WHAT CAN YOU DO
Import an existing, pre-trained model.
Re-train an imported model (transfer learning)
WHAT CAN YOU DO
Import an existing, pre-trained model.
Re-train an imported model (transfer learning)
Define, train and run models entirely in the browser.
TOOLS
DEMOS & CODE
5
USING A PRE-TRAINED MODEL
RECYCLING
CODE SAMPLE
import * as cocoSsd from "@tensorflow-models/coco-ssd";
import * as tf from '@tensorflow/tfjs-core';
let model = await cocoSsd.load();
let webcam = document.getElementById('webcam');
const webcamImage = tf.browser.fromPixels(webcam);
const batchedImage = webcamImage.expandDims(0);
const processedImage =
batchedImage.toFloat().div(tf.scalar(127)).sub(tf.scalar(1));
let predictions = await model.detect(processedImage);
return predictions;
CODE SAMPLE
import * as cocoSsd from "@tensorflow-models/coco-ssd";
import * as tf from '@tensorflow/tfjs-core';
let model = await cocoSsd.load();
let webcam = document.getElementById('webcam');
const webcamImage = tf.browser.fromPixels(webcam);
const batchedImage = webcamImage.expandDims(0);
const processedImage =
batchedImage.toFloat().div(tf.scalar(127)).sub(tf.scalar(1));
let predictions = await model.detect(processedImage);
return predictions;
CODE SAMPLE
import * as cocoSsd from "@tensorflow-models/coco-ssd";
import * as tf from '@tensorflow/tfjs-core';
let model = await cocoSsd.load();
let webcam = document.getElementById('webcam');
const webcamImage = tf.browser.fromPixels(webcam);
const batchedImage = webcamImage.expandDims(0);
const processedImage =
batchedImage.toFloat().div(tf.scalar(127)).sub(tf.scalar(1));
let predictions = await model.detect(processedImage);
return predictions;
CODE SAMPLE
import * as cocoSsd from "@tensorflow-models/coco-ssd";
import * as tf from '@tensorflow/tfjs-core';
let model = await cocoSsd.load();
let webcam = document.getElementById('webcam');
const webcamImage = tf.browser.fromPixels(webcam);
const batchedImage = webcamImage.expandDims(0);
const processedImage =
batchedImage.toFloat().div(tf.scalar(127)).sub(tf.scalar(1));
let predictions = await model.detect(processedImage);
return predictions;
TRANSFER LEARNING
THE TEACHABLE KEYBOARD
CODE SAMPLE
import * as mobilenetModule from '@tensorflow-models/mobilenet';
import * as tf from '@tensorflow/tfjs';
import * as knnClassifier from '@tensorflow-models/knn-classifier';
// Number of classes to classify
const NUM_CLASSES = 4;
// Webcam Image size. Must be 227.
const IMAGE_SIZE = 227;
// K value for KNN
const TOPK = 10;
const classes = ["Right", "Left", "Down", "Neutral"];
CODE SAMPLE
import * as mobilenetModule from '@tensorflow-models/mobilenet';
import * as tf from '@tensorflow/tfjs';
import * as knnClassifier from '@tensorflow-models/knn-classifier';
// Number of classes to classify
const NUM_CLASSES = 4;
// Webcam Image size. Must be 227.
const IMAGE_SIZE = 227;
// K value for KNN
const TOPK = 10;
const classes = ["Right", "Left", "Down", "Neutral"];
CODE SAMPLE
const knn = knnClassifier.create();
const mobilenet = await mobilenetModule.load();
const image = tf.browser.fromPixels(video);
let logits = mobilenet.infer(image, 'conv_preds');
knn.addExample(logits, training);
const res = await knn.predictClass(logits, TOPK);
console.log(classes[res.classIndex]) // ‘Right’, ‘Left’, …
// Dispose image when done
image.dispose();
CODE SAMPLE
const knn = knnClassifier.create();
const mobilenet = await mobilenetModule.load();
const image = tf.browser.fromPixels(video);
let logits = mobilenet.infer(image, 'conv_preds');
knn.addExample(logits, training);
const res = await knn.predictClass(logits, TOPK);
console.log(classes[res.classIndex]) // ‘Right’, ‘Left’, …
// Dispose image when done
image.dispose();
CODE SAMPLE
const knn = knnClassifier.create();
const mobilenet = await mobilenetModule.load();
const image = tf.browser.fromPixels(video);
let logits = mobilenet.infer(image, 'conv_preds');
knn.addExample(logits, training);
const res = await knn.predictClass(logits, TOPK);
console.log(classes[res.classIndex]) // ‘Right’, ‘Left’, …
// Dispose image when done
image.dispose();
CODE SAMPLE
const knn = knnClassifier.create();
const mobilenet = await mobilenetModule.load();
const image = tf.browser.fromPixels(video);
let logits = mobilenet.infer(image, 'conv_preds');
knn.addExample(logits, training);
const res = await knn.predictClass(logits, TOPK);
console.log(classes[res.classIndex]) // ‘Right’, ‘Left’, …
// Dispose image when done
image.dispose();
CODE SAMPLE
const knn = knnClassifier.create();
const mobilenet = await mobilenetModule.load();
const image = tf.browser.fromPixels(video);
let logits = mobilenet.infer(image, 'conv_preds');
knn.addExample(logits, training);
const res = await knn.predictClass(logits, TOPK);
console.log(classes[res.classIndex]) // ‘Right’, ‘Left’, …
// Dispose image when done
image.dispose();
CODE SAMPLE
const knn = knnClassifier.create();
const mobilenet = await mobilenetModule.load();
const image = tf.browser.fromPixels(video);
let logits = mobilenet.infer(image, 'conv_preds');
knn.addExample(logits, training);
const res = await knn.predictClass(logits, TOPK);
console.log(classes[res.classIndex]) // ‘Right’, ‘Left’, …
// Dispose image when done
image.dispose();
EVERYTHING!
THE WILLY DETECTOR
PROCESS
Hand-drawn
Quickdraw dataset
Data collection
PROCESS
200x200
Data processing
28x28
PROCESS
Splitting training / test set
Training data - 80%
Test data - 20%
PROCESS
Choose algorithm
Convolutional Neural Network (CNN) using Keras
PROCESS
Final steps
Parameter tuning
PROCESS
Final steps
Parameter tuning
Train
PROCESS
Final steps
Parameter tuning
Train
Prediction
CODE SAMPLE
getTrainData() {
const xs = tf.tensor4d(this.trainImages, [NUM_TRAIN_ELEMENTS, IMAGE_H, IMAGE_W, 1]);
const labels = tf.tensor2d(this.trainLabels, [this.trainLabels.length / NUM_CLASSES, NUM_CLASSES]);
return {xs, labels};
}
getTestData() {
let xs = tf.tensor4d(this.testImages,[this.testImages.length / IMAGE_SIZE, IMAGE_H, IMAGE_W, 1]);
let labels = tf.tensor2d(this.testLabels, [this.testLabels.length / NUM_CLASSES, NUM_CLASSES]);
return {xs, labels};
}
CODE SAMPLE
const model = tf.sequential();
model.add(tf.layers.conv2d({
inputShape: [28, 28, 1],
kernelSize: 3,
filters: 16,
activation: 'relu'
}));
model.add(tf.layers.maxPooling2d({poolSize: 2, strides: 2}));
model.add(tf.layers.conv2d({kernelSize: 3, filters: 32, activation: 'relu'}));
model.add(tf.layers.flatten({}));
model.add(tf.layers.dense({units: 64, activation: 'relu'}));
model.add(tf.layers.dense({units: NUM_CLASSES, activation: 'softmax'}));
return model;
CODE SAMPLE
const trainData = getTrainData();
await model.fit(trainData.xs, trainData.labels, {
batchSize, // 2
validationSplit, // 0.15
epochs: trainEpochs, // 3
});
const examples = getTestData();
const output = model.predict(examples.xs);
const labels = Array.from(examples.labels.argMax().dataSync());
const predictions = Array.from(output.argMax().dataSync());
return {predictions, labels};
LIMITS
6
LIMITS
LIMITS
LIMITS
LIMITS
LIMITS
ALGORITHMS ARE NOT BIASED,
WE ARE.
ETHICS
ADVERSARIAL EXAMPLES
“[...] deep-learning models are also brittle. Because an image recognition system relies only on pixel patterns rather than a deeper conceptual understanding of what it sees, it’s easy to trick the system into seeing something else entirely—just by disturbing the patterns in the right way.”
RESOURCES
7
TOOLS
Udacity - Intro to Machine Learning
Google - Machine learning crash course
Neural networks and Deep Learning - free e-book
Machine Learning 101 - slide deck
The Ethics and governance of Artificial Intelligence
Deep Learning in JS - Ashi Krishnan
Play street fighter with body movements with Arduino & Tensorflow.js
USEFUL LINKS
Core50 - Continuous object detection
UCI Machine Learning repository
Pre-trained models for Tensorflow.js
DATASETS
MACHINE LEARNING FOR FRONT-END DEVELOPERS
THANK YOU!
@devdevcharlie