DIY Deep Learning for Vision:
a Hands-On Tutorial with Caffe
Evan Shelhamer, Jeff Donahue, Jon Long, Yangqing Jia, and Ross Girshick
Look for further details in the �outline notes
Tutorial Schedule�Caffe tour and latest roast
Caffe Tour
Latest Roast
Why Deep Learning? �End-to-End Learning for Many Tasks
What is Deep Learning?
Compositional Models
Learned End-to-End
Hierarchy of Representations
concrete
abstract
learning
figure credit Yann LeCun, ICML ‘13 tutorial
What is Deep Learning?
Compositional Models
Learned End-to-End
figure credit Yann LeCun, ICML ‘13 tutorial
Back-propagation jointly learns�all of the model parameters to
optimize the output for the task.
All in a day’s work with Caffe
What is Caffe?
Open framework, models, and worked examples�for deep learning
9
Prototype
Train
Deploy
What is Caffe?
Open framework, models, and worked examples�for deep learning
10
Prototype
Train
Deploy
Caffe is a Community project pulse
11
Reference Models
Caffe offers the
so you can start right away.
The BVLC models are licensed for unrestricted use.
The community shares models in our Model Zoo.
GoogLeNet: ILSVRC14 winner
Open Model Collection
The Caffe Model Zoo open collection of deep models to share innovation
�helps disseminate and reproduce research
bundled tools for loading and publishing models
Share Your Models! with your citation + license of course
13
Brewing by the Numbers...
Speed with Krizhevsky's 2012 model:
9k lines of C++ code (20k with tests)
14
CAFFE
EXAMPLES + APPLICATIONS
Share a Sip of Brewed Models
demo code open-source and bundled
Scene Recognition http://places.csail.mit.edu/
B. Zhou et al. NIPS 14
Visual Style Recognition
Karayev et al. Recognizing Image Style. BMVC14. Caffe fine-tuning example.
Demo online at http://demo.vislab.berkeleyvision.org/ (see Results Explorer).
[ Image-Style]
Object Detection
R-CNNs: Region-based Convolutional Networks
Ross Girshick, Shaoqing Ren, Kaiming He, Jian Sun
papers + code online
Pixelwise Prediction
Fully convolutional networks for pixel prediction�in particular semantic segmentation
20
Applications
Jon Long* & Evan Shelhamer*,
Trevor Darrell. CVPR’15
CVPR'15 paper and code + models
Visual Sequence Tasks
21
Jeff Donahue et al. CVPR’15
Recurrent Networks for Sequences
Recurrent Nets and Long Short Term Memories (LSTM) �are sequential models
learned by backpropagation through time
22
LRCN: Long-term Recurrent Convolutional Network
LRCN:
recurrent + convolutional for visual sequences
CVPR'15 paper and code + models
Deep Visuomotor Control
23
Sergey Levine* & Chelsea Finn*,�Trevor Darrell, and Pieter Abbeel
example experiments
feature visualization
Deep Visuomotor Control Architecture
24
Sergey Levine* & Chelsea Finn*,�Trevor Darrell, and Pieter Abbeel
Embedded Caffe
Caffe runs on embedded CUDA hardware and mobile devices
25
CUDA Jetson TX1, TK1
Caffeinated Companies
26
… startups, big companies, more ...
Caffe at Facebook
27
Automatic Alt Text
recognize photo content�for accessibility
[example credit Facebook]
On This Day
highlight content
Caffe at Pinterest
28
[example credit Andrew Zhai, Pinterest]
Caffe at Adobe
29
Photoshop Type Similarity
catalogue typefaces automatically
Caffe at Yahoo! Japan
30
News Image Recommendation
select and crop images for news
Classification
instant recognition the Caffe way
Convolutional Network
Convolutional Networks: 1989
LeNet: a layered model composed of convolution and subsampling operations followed by a holistic representation and ultimately a classifier for handwritten digits. [ LeNet ]
Convolutional Nets: 2012
AlexNet: a layered model composed of convolution, subsampling, and further operations followed by a holistic representation and all-in-all a landmark classifier on
ILSVRC12. [ AlexNet ]
+ data
+ gpu
+ non-saturating nonlinearity
+ regularization
Convolutional Nets: 2014
ILSVRC14 Winners: ~6.6% Top-5 error
+ depth
+ data
+ dimensionality reduction
Learning LeNet
back to the future of visual recognition
Deep Learning, as it is executed...
What should a framework handle?
Compositional Models
Decompose the problem and code!
End-to-End Learning
Configure and solve!
Many Architectures and Tasks
Define, experiment, and extend!
Net
name: "dummy-net"
layer { name: "data" …}
layer { name: "conv" …}
layer { name: "pool" …}
… more layers …
layer { name: "loss" …}
LogReg ↑
LeNet →
ImageNet, Krizhevsky 2012 →
Forward / Backward the essential Net computations
Caffe models are complete machine learning systems for inference and learning.
The computation follows from the model definition. Define the model and run.
DAG
Many current deep models
have linear structure
but Caffe nets can have any directed acyclic graph (DAG) structure.
Define bottoms and tops�and Caffe will connect the net.
LRCN joint vision-sequence model
GoogLeNet Inception Module
SDS two-stream net
Layer Protocol
Setup: run once for initialization.
Forward: make output given input.
Backward: make gradient of output
- w.r.t. bottom
- w.r.t. parameters (if needed)
Reshape: set dimensions.
Compositional Modeling�The Net’s forward and backward passes are composed of the layers’ steps.
Layer Protocol
== Class Interface
Define a class in C++ or Python to extend Layer.
Include your new layer type in a network and keep brewing.
layer {
type: "Python"�python_param {
module: "layers"
layer: "EuclideanLoss"
} }
Blob
name: "conv1"
type: CONVOLUTION
bottom: "data"
top: "conv1"
… definition …
Data
Number x K Channel x Height x Width
256 x 3 x 227 x 227 for ImageNet train input
Blobs are N-D arrays for storing and communicating information.
top �blob
bottom blob
Parameter: Convolution Weight
N Output x K Input x Height x Width
96 x 3 x 11 x 11 for CaffeNet conv1
Parameter: Convolution Bias
96 x 1 x 1 x 1 for CaffeNet conv1
Blob
Blobs provide a unified memory interface.
Reshape(num, channel, height, width)
- declare dimensions
- make SyncedMem -- but only lazily allocate
cpu_data(), mutable_cpu_data()
- host memory for CPU mode
gpu_data(), mutable_gpu_data()
- device memory for GPU mode
{cpu,gpu}_diff(), mutable_{cpu,gpu}_diff()
- derivative counterparts to data methods
- easy access to data + diff in forward / backward
SyncedMem
allocation + communication
Loss
Classification
SoftmaxWithLoss
HingeLoss
Linear Regression
EuclideanLoss
Attributes / Multiclassification
SigmoidCrossEntropyLoss
Others…
New Task
NewLoss
What kind of model is this?
Define the task by the loss.
loss (LOSS_TYPE)
Protobuf Model Format
message ConvolutionParameter {
// The number of outputs for the layer
optional uint32 num_output = 1;
// whether to have bias terms
optional bool bias_term = 2 [default = true];
}
name: "conv1"
type: "Convolution"
bottom: "data"
top: "conv1"
convolution_param {
num_output: 20
kernel_size: 5
stride: 1
weight_filler {
type: "xavier"
}
}
Model Zoo Format
Gists on github hold model definition, license, url for weights, and hash of Caffe commit that guarantees compatibility.
Solving: Training a Net
Optimization like model definition is configuration.
train_net: "lenet_train.prototxt"
base_lr: 0.01
momentum: 0.9
weight_decay: 0.0005
max_iter: 10000
snapshot_prefix: "lenet_snapshot"
All you need to run things on the GPU.
> caffe train -solver lenet_solver.prototxt -gpu 0
Stochastic Gradient Descent (SGD) + momentum ·
Adaptive Gradient (ADAGRAD) · Nesterov’s Accelerated Gradient (NAG)
Solver Showdown: MNIST Autoencoder
I0901 13:36:30.007884 24952 solver.cpp:232] Iteration 65000, loss = 64.1627�I0901 13:36:30.007922 24952 solver.cpp:251] Iteration 65000, Testing net (#0) # train set�I0901 13:36:33.019305 24952 solver.cpp:289] Test loss: 63.217�I0901 13:36:33.019356 24952 solver.cpp:302] Test net output #0: cross_entropy_loss = 63.217 (* 1 = 63.217 loss)�I0901 13:36:33.019773 24952 solver.cpp:302] Test net output #1: l2_error = 2.40951
AdaGrad
SGD
Nesterov
I0901 13:35:20.426187 20072 solver.cpp:232] Iteration 65000, loss = 61.5498�I0901 13:35:20.426218 20072 solver.cpp:251] Iteration 65000, Testing net (#0) # train set�I0901 13:35:22.780092 20072 solver.cpp:289] Test loss: 60.8301�I0901 13:35:22.780138 20072 solver.cpp:302] Test net output #0: cross_entropy_loss = 60.8301 (* 1 = 60.8301 loss)�I0901 13:35:22.780146 20072 solver.cpp:302] Test net output #1: l2_error = 2.02321
I0901 13:36:52.466069 22488 solver.cpp:232] Iteration 65000, loss = 59.9389�I0901 13:36:52.466099 22488 solver.cpp:251] Iteration 65000, Testing net (#0) # train set�I0901 13:36:55.068370 22488 solver.cpp:289] Test loss: 59.3663�I0901 13:36:55.068410 22488 solver.cpp:302] Test net output #0: cross_entropy_loss = 59.3663 (* 1 = 59.3663 loss)�I0901 13:36:55.068418 22488 solver.cpp:302] Test net output #1: l2_error = 1.79998
Weight Sharing
layer: {
name: 'innerproduct1'
type: INNER_PRODUCT
inner_product_param {
num_output: 10
bias_term: false
weight_filler {
type: 'gaussian'
std: 10
}
}
param: 'sharedweights'
bottom: 'data'
top: 'innerproduct1'
}
layer: {
name: 'innerproduct2'
type: INNER_PRODUCT
inner_product_param {
num_output: 10
bias_term: false
}
param: 'sharedweights'
bottom: 'data'
top: 'innerproduct2'
}
Recipe for Brewing
Brewing Models
from logistic regression to non-linearity�see notebook
Take a pre-trained model and fine-tune to new tasks �[DeCAF] [Zeiler-Fergus] [OverFeat]
Dogs vs.
Cats
top 10 in�10 minutes
© kaggle.com
Your Task
Style
Recognition
Lots of Data
ImageNet
image by Andrej Karpathy
From ImageNet to Style
Simply change a few lines in the model definition
layer {� name: "data"� type: "Data"
data_param {
source: "ilsvrc12_train_lmdb"
mean_file: "../../data/ilsvrc12"
...
}
...
}
...
layer {� name: "fc8"� type: "InnerProduct"
inner_product_param {
num_output: 1000
...
}
}
layer {� name: "data"� type: "Data"
data_param {
source: "style_train_lmdb"
mean_file: "../../data/ilsvrc12"
...
}
...
}
...
layer {� name: "fc8-style"� type: "InnerProduct"
inner_product_param {
num_output: 20
...
}
}
Input:
A different source
Last Layer:
A different classifier
new name =�new params
From ImageNet to Style
> caffe train -solver models/finetune_flickr_style/solver.prototxt
-weights bvlc_reference_caffenet.caffemodel
Step-by-step in pycaffe:
pretrained_net = caffe.Net(
"net.prototxt", "net.caffemodel")
solver = caffe.SGDSolver("solver.prototxt")
solver.net.copy_from(pretrained_net)
solver.solve()
Fine-tuning
transferring features to style recognition
When to Fine-tune?
A good first step
State-of-the-art results in
[Zeiler-Fergus]
Fine-tuning Tricks
Learn the last layer first
Reduce the learning rate
Do net surgery see notebook on editing model parameters
Transferability of Features
Yosinski et al. NIPS 2014
After fine-tuning
P. Agarwal et al. ECCV 14
Editing model parameters
how to do net surgery to set custom weights
Up Next The Latest Roast
Detection
Fast R-CNN
Faster R-CNN
Ross Girshick, Shaoqing Ren, Kaiming He, Jian Sun
Pixelwise Prediction
Fully convolutional networks for pixel prediction
applied to semantic segmentation
Further applications
Jon Long* & Evan Shelhamer*,
Trevor Darrell
CVPR15 arXiv and reference models + code
Sequences
Recurrent Net and Long Short Term Memory LSTM �are sequential models
learned by backpropagation through time.
Jeff Donahue et al.
LRCN: Long-term Recurrent Convolutional Network
CVPR15 arXiv and project site
Framework Future
1.0 is coming stability, documentation, packaging
Performance Tuning for GPU (cuDNN v5) and CPU (nnpack)
In-progress Ports for OpenCL and Windows
Halide interface for prototyping and experimenting
Widening the Circle continued and closer collaborative development
66
Next Steps
Now you’ve seen the progress made with�DIY deep learning and the democratization of models
�Next Up:
67
Join the caffe-users mailing list
Check out Caffe on github
Run Caffe through Docker�and NVIDIA Docker for GPU
Help Brewing
Documentation
Modeling, Usage, and �Installation
Convolutional Nets
68
Thanks to the Caffe Crew
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Jonathan Long,�Sergey Karayev, Ross Girshick, Sergio Guadarrama, Ronghang Hu, Trevor Darrell
and our open source contributors!
69
...plus the cold-brew
Acknowledgements
70
Thank you to the Berkeley Vision and Learning Center and its Sponsors
Thank you to NVIDIA
for GPUs, cuDNN collaboration,�and hands-on cloud instances
Thank you to our 150+
open source contributors
and vibrant community!
Thank you to A9 and AWS
for a research grant for Caffe dev and reproducible research
References
[ DeCAF ] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. ICML, 2014.
[ R-CNN ] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. CVPR, 2014.
[ Zeiler-Fergus ] M. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. ECCV, 2014.
[ LeNet ] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. IEEE, 1998.
[ AlexNet ] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. NIPS, 2012.
[ OverFeat ] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. ICLR, 2014.
[ Image-Style ] S. Karayev, M. Trentacoste, H. Han, A. Agarwala, T. Darrell, A. Hertzmann, H. Winnemoeller. Recognizing Image Style. BMVC, 2014.
[ Karpathy14 ] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. CVPR, 2014.
[ Sutskever13 ] I. Sutskever. Training Recurrent Neural Networks.�PhD thesis, University of Toronto, 2013.
[ Chopra05 ] S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity metric discriminatively, with application to face verification. CVPR, 2005.
END
MORE DETAILS
Why Deep Learning? �The Unreasonable Effectiveness of Deep Features
Classes separate in the deep representations and transfer to many tasks.�[DeCAF] [Zeiler-Fergus]
Why Deep Learning? �The Unreasonable Effectiveness of Deep Features
Rich visual structure of features deep in hierarchy.
[R-CNN]
[Zeiler-Fergus]
Maximal activations of pool5 units
conv5 DeConv visualization
Why Deep Learning? �The Unreasonable Effectiveness of Deep Features
[Zeiler-Fergus]
1st layer filters
image patches that strongly activate 1st layer filters