Run Deep Learning Models in the Browser with JavaScript and ConvNetJS

Christoph Körner

​

​

​

Slides available via http://bit.ly/jazoon2017-dl

About me

• Big Data Tech Lead at T-Mobile Austria
• Visual Computing at Vienna University of Technology
• Creator of/Contributor to CaffeJS
• Author of Data Visualizations with D3 and AngularJS
• Author of Learning Responsive Data Visualization
• Organizer of Vienna Kaggle Meetup

​

​

Agenda

1. Why JavaScript might be a thing for Deep Learning
2. Intro to Neural Nets and Deep Learning
3. Deep Learning and JavaScript
4. More Demos

Y THO

Source: twitter.com/lazygamereviews

Why JavaScript might be a thing for DL

• Runs in the Browser (and hence on almost any device)
• Runs on the Server
• Ridiculously easy Debugging
• Interactive Visualization (of activations, gradients, etc.)

Besides Education

• Access to a LOT of data on the client side
• Models built on Cursor movement, typing speed, etc.
• Massive parallelization
• Run backward passes on all your clients and collect gradients
• Privacy sensitive applications
• Run forward passes on the client only

Runs in the Browser

Source: chaosmail.github.io/caffejs

...and hence almost anywhere

Ridiculously easy Debugging

Source: chaosmail.github.io/caffejs

Interactive Visualization

Source: ConvNetJS - MNIST

Intro to Neural Nets

(Supervised Classification)

Pedestrian or cyclist?

Dataset

x₀

x₁

Annotated Dataset

Pedestrian

Cyclist

x₀

x₁

Random Separation Line

Pedestrian

Cyclist

x₀

x₁

Line Equation

y = w * x + b

Line Equation

y = w * x + b

Distance from the line

weights

bias

Input data

Line Equation

x

y

b

w

Line Equation

x₀

y

b

w

x₁

Line Equation (simplified)

x₀

y

w

x₁

Let’s classify the samples

Pedestrian

Cyclist

x₀

x₁

Activation function

+1

-1

Pedestrian

Cyclist

x₀

x₁

Activation function

x₀

y

w

x₁

Step function:

map to -1 and +1

Neuron

x₀

y

x₁

How good are the predicted classes?

Pedestrian

Cyclist

x₀

x₁

+1

-1

Error/Loss function

Wrong side!

Pedestrian

Cyclist

x₀

x₁

+1

-1

Error/Loss function

x₀

y

w

x₁

Compute Error

y_true

Optimization (e.g. Gradient Descent)

x₀

w

x₁

Update weights

y

y_true

Optimization (e.g. Gradient Descent)

Pedestrian

Cyclist

x₀

x₁

Optimization (e.g. Gradient Descent)

Pedestrian

Cyclist

x₀

x₁

Optimization (e.g. Gradient Descent)

Pedestrian

Cyclist

x₀

x₁

Optimization (e.g. Gradient Descent)

Pedestrian

Cyclist

x₀

x₁

Wrong side!

Multiple Neurons

Pedestrian

Cyclist

+1

-1

+1

-1

x₀

x₁

Multiple Neurons

x₀

y₀

x₁

y₁

Multiple Neurons

Pedestrian

Cyclist

+1

-1

x₀

x₁

Multiple Layers

x₀

x₁

y

Summary (Neural Nets)

• Neurons stacked in Layers
• Line equation (Hyperplane)
• Activation function
• Outputs class (score) in the end
• Optimization
• Compute error (with ground truth)
• Update weights
• Iterate

Summary (Neural Nets)

• Neurons stacked in Layers
• Line equation (Hyperplane)
• Activation function
• Outputs class (score) in the end
• Optimization
• Compute error (with ground truth)
• Update weights
• Iterate

• Training
• Backward Pass
• Back-Propagation

• Inference
• Forward Pass

Summary (Neural Nets)

x₀

x₁

y

Inference

Back-Propagation

Demo: Tensorflow Playground

Source: playground.tensorflow.org

Ingredients for Deep Learning

• More (efficient) Layers
• Convolutions
• Pooling
• Batchnorm, Concat, Skip, etc.
• Deep and wide models
• A lot of data

Fully Connected Layer

x₀

x₁

y

x₂

x₃

Usually very large! e.g. 3x227x227

Hence a large number of params e.g. 3x227x227 * 2

Convolution (weight sharing)

x₀

x₁

y

x₂

x₃

Convolution (weight sharing)

x₀

x₁

y

x₂

x₃

Convolution (weight sharing)

x₀

x₁

y

x₂

x₃

Demo: MNIST Classification (Training)

Source: ConvNetJS - MNIST

Demo: Deep Learning Models

• AlexNet (2012)
• 60M params ~ 240MB
• 15.4% ImageNet Top-5 Error
• GoogLeNet (2014)
• 7M params ~ 30MB
• 6.7% ImageNet Top-5 Error
• SqueezeNet (2016)
• 1M params ~ 5MB
• 17.5% ImageNet Top-5 Error

Source: chaosmail.github.io/caffejs

Demo: DeepDream

Source: Google DeepDream, chaosmail.github.io/caffejs

Demo: DeepDream

Source: Google DeepDream, chaosmail.github.io/caffejs

Many pretrained models available

• For Tasks:
• Image Classification
• Image Segmentation
• Object Localization and Detection
• Word embedding (word2vec)
• … many more
• For Frameworks:
• Caffe: github.com/BVLC/caffe/wiki/Model-Zoo
• Keras/Tensorflow: keras.io/applications/
• Torch: github.com/torch/torch7/wiki/ModelZoo

Summary (Deep Learning)

• Use mostly convolutions (weight sharing)
• Deep networks for learning complex features
• Many pretrained models are available
• Caffe models: BAIR license for unrestricted use

JavaScript Ingredients for Deep Learning

• TypedArrays and ArrayBuffer
• GPU support via WebGL
• Multithreading via WebWorkers
• Rich Media APIs
• WebRTC (Web Real-Time Communications)
• WebAudio & WebMidi

Challenges in JavaScript (in the Browser)

• Network load
• Memory consumption
• No CUDA, requires Textures in WebGL�
• Small models are preferred

JavaScript Frameworks for Deep Learning

It’s time for some more demos

Teachable Machine

Authors: Støj, Use All Five, Creative Lab and PAIR teams at Google; Source: teachablemachine.withgoogle.com

Fast Neural Style Transfer

Authors: Reiichiro Nakano; Source: reiinakano.github.io/fast-style-transfer-deeplearnjs

Sentiment Classification

Authors: Leon Chen (MD.ai); Source: transcranial.github.io/keras-js/#/imdb-bidirectional-lstm

RNN for Melody Creation

Authors: Magenta (a Google Brain project); Source: deeplearnjs.org/demos/performance_rnn

Take-home Messages

• Already today, you can run many pretrained models and train models in the browser for image, audio, text, etc.
• Keep an eye on the model size
• Avoid Fully Connected layers
• Embed wide modules (such as Fire or Inception)
• Use GPU acceleration if possible

One more thing..

It’s an exciting time for JavaScript developers interested in AI and DL!

Thanks!

​

Slides are available:

bit.ly/jazoon2017-dl

​

​