ZNN - A CPU Implementation of Convolutional Neural Networks for Deep Learning
Aleks Zlateski
1/23/2015
Motivation
Why do we need ZNN?
Why do we need fast convolutional networks?
Why do we need ZNNs?
Why do we need ZNNs?
Seung Lab
Members
We have a EM microscope!
Obtain high resolution EM images
3D volume of neural tissue
So?
Ashwin Vishwanathan
3D reconstruct neurons...
Kim at al.
3D reconstruct neurons...
Morphology of different cell types
Rendering by
Alex Norton
Synapses
Rendering by
Alex Norton
Understand the brain a bit better
Rendering by
Alex Norton
Rendering by
Alex Norton
3D reconstruct neurons…
Kim at al.
3D reconstruct neurons…
How?
Kim at al.
Boundaries to segmentation
Main challenge: find the boundaries!
Convolutional Neural Networks
Convolutional Neural Networks
Graph Representation
Convolutional Neural Networks
Graph Representation
Convolutional Neural Networks
Graph Representation
Convolutional Neural Networks
Graph Representation
Convolutional Neural Networks
Graph Representation
[0,1]
Convolutional Neural Networks
Graph Representation
[0,1]
Field of view
Convolutional Neural Networks
raph Representation
Training Neural Networks
Prepare a training set
Training Neural Networks
Backpropagation using gradient descent
Phases
Training Neural Networks
Backpropagation using gradient descent
Phases
[0,1]
Training Neural Networks
Backpropagation using gradient descent
Phases
Error
Training Neural Networks
Backpropagation using gradient descent
Phases
Convolutional Neural Networks
Graph Representation
[0,1]
Field of view
Field of view size matters!
Field of view size matters!
Increasing the Field of View
Deep Neural Networks
Deep Neural Networks
TIM the MIT beaver
How can we make ZNN fast?
Faster convolution using FFTs
Direct Convolution
O(N3*M3)
Faster convolution using FFTs
Direct Convolution
O(N3*M3)
FFT Convolution
O(N3logN)
Faster for large enough M
(larger filters)
Faster convolution using FFTs
Direct Convolution
O(N3*M3)
FFT Convolution
O(N3logN)
Slower for some M
Multiple convolutions still faster!
Faster convolution using FFTs
X*Y Convolutions
X Y
Faster convolution using FFTs
X*Y Convolutions
X Y
Faster convolution using FFTs
X*Y Convolutions
X Y
Faster convolution using FFTs
X*Y Convolutions
X Y
Faster convolution using FFTs
X*Y Convolutions
X Y
Faster convolution using FFTs
X Y
ZNN Parallelization Ideas
ZNN Parallelization Ideas
ZNN Parallelization Model
Convolutional Neural Network
Forward Pass Tasks - Type 1
Convolutional Neural Network
Forward Pass Tasks - Type 2
Convolutional Neural Network
Backward Pass Tasks - Type 1
Convolutional Neural Network
Backward Pass Tasks - Type 2
Convolutional Neural Network
Weight Update Tasks - Type 2
ZNN Parallelization Model
Zonvolutional Neural Network
Forward Pass - Scheduling Tasks
Zonvolutional Neural Network
Forward Pass - Scheduling Tasks
Zonvolutional Neural Network
Forward Pass - Scheduling Tasks
Zonvolutional Neural Network
Forward Pass - Scheduling Tasks
Zonvolutional Neural Network
Forward Pass - Scheduling Tasks
Zonvolutional Neural Network
Backward Pass - Scheduling Tasks
Zonvolutional Neural Network
Weight Update - Scheduling Tasks
Zonvolutional Neural Network
Weight Update
Zonvolutional Neural Network
Further Optimizations
Some Benchmarks
Graph Representation
Some Benchmarks
Scalability Test - 4 layer 16 width network
Some Benchmarks
Compared to CNS
Updates per
second
CNS
ZNN
Some Benchmarks
Compared to CNS
CNS
ZNN
Updates per
second
Some Benchmarks
Compared to CNS
CNS
ZNN
Updates per
second
Some Benchmarks
Compared to Theano GPU
CNS
ZNN
Seconds per
update
Why is ZNN is so fast?
When everyone is says we should use GPUs?
CPUs vs GPUs
CPUs vs GPUs
CPUs vs GPUs
CPUs vs GPUs
CPUs vs GPUs
GPUs still faster!
CPUs vs GPUs
GPU limitations
ZNN loves deep learning
ZNN is heavily used in-house
KisukNetTM v1 (kisuklee@mit.edu)
KisukNetTM v2 (kisuklee@mit.edu)
KisukNetTM v3 (kisuklee@mit.edu)
KisukNetTM v73 (kisuklee@mit.edu)
Ashwin Vishwanathan
2500 pxl (17.5 μm)
2500 pxl (17.5 μm)
Kisuk
Lee
ZNN now
ZNN in the future
ZNN in the future
ZNN in the future
1. Graphics
ZNN in the future
1. Graphics
2. Graphics
ZNN in the future
1. Graphics
2. Graphics
…
N-1. Graphics
ZNN in the future
1. Graphics
2. Graphics
…
N-1. Graphics
N. GPU Compute
Why should you use ZNNs
Questions?