Introduction to computer vision 13

Jean Ponce

jean.ponce@ens.fr

​

Zuhaib Akhtar za2023@nyu.edu

Ayush Jain aj3152@nyu.edu

​

​

​

Slides will be available after classes

Deep learning

​

• Representation learning

​

• History of neural networks

​

• Training

​

• CNNs

​

Image categorization as

representation learning

Image “space”

Feature (Hilbert) space

θ

θ

θ

Deep learning

Layer 1

Layer 2

Layer n

Linear head

Learned

representation

Traditional Recognition Approach

Hand-designed�feature extraction

Trainable�classifier

Image/ Video

Pixels

Object�Class

What about learning the features?

• Learn a feature hierarchy all the way from pixels to classifier
• Each layer extracts features from the output of previous layer
• Train all layers jointly

Layer 1

Layer 2

Layer 3

Simple �Classifier

Image/ Video

Pixels

“Shallow” vs. “deep” architectures

Hand-designed�feature extraction

Trainable�classifier

Image/ Video

Pixels

Object�Class

Layer 1

Layer N

Simple classifier

Object Class

Image/ Video

Pixels

Traditional recognition: “Shallow” architecture

Deep learning: “Deep” architecture

…

Brief history of Neural Networks

• Perceptron algorithm [Rosenblatt, 57]

Output:

bias

(Source Wikipedia)

Brief history of Neural Networks

• Perceptron algorithm [Rosenblatt, 57]
• Multi-layer perceptron with hidden layers

Inspiration: Neuron cells

Hubel/­Wiesel Architecture

• D. Hubel and T. Wiesel (1959, 1962, Nobel Prize 1981)
• Visual cortex consists of a hierarchy of simple, complex, and hyper-complex cells

Brief history of Neural Networks

• Perceptron algorithm [Rosenblatt, 57]
• Multi-layer perceptron with hidden layers
• Shift-invariant network inspired by Hubel & Wiesel study of visual cortex “Neocognitron” [Fukushima 1980]

Brief history of Neural Networks

• Perceptron algorithm [Rosenblatt, 57]
• Multi-layer perceptron with hidden layers
• Shift-invariant network inspired by Hubel & Wiesel study of visual cortex “Neocognitron” [Fukushima 1980]
• Training of multi-layer networks with “backpropagation”

Convolutional Neural Networks (CNN, Convnet)

• Neural network with specialized connectivity structure
• Stack multiple stages of feature extractors
• Higher stages compute more global, more invariant features
• Classification layer at the end

​

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86(11): 2278–2324, 1998.

Parametric supervised learning

(Slides borrowed from S. Lazebnik and M. Trager)

Neural networks

Feedforward neural network

Nonlinearity

Convolutional Neural Networks (CNN, Convnet)

• Feed-forward feature extraction:
1. Convolve input with learned filters
2. Non-linearity
3. Spatial pooling
4. Normalization
• Supervised training of convolutional �filters by back-propagating �classification error

Input Image

Convolution (Learned)

Non-linearity

Spatial pooling

Normalization

Feature maps

Feedforward neural networks for images

image

Fully connected layer

Source: S. Lazebnik

CNNs: Neural networks for images

image

Convolutional layer

Source: S. Lazebnik

CNNs: Neural networks for images

image

feature map

learned weights

Convolutional layer

Source: S. Lazebnik

CNNs: Neural networks for images

image

feature map

learned weights

Convolutional layer

Source: S. Lazebnik

Convolution as feature extraction

Input

Feature Map

...

Source: S. Lazebnik

Neural networks for images

image

feature map

learned weights

Convolutional layer

Source: S. Lazebnik

Neural networks for images

image

next layer

Convolutional layer

+ ReLU

Source: S. Lazebnik

Key operations in a CNN

Source: R. Fergus, Y. LeCun

Key operations in a CNN

Source: R. Fergus, Y. LeCun

Rectified Linear Unit (ReLU)

Key operations in a CNN

Max

Source: R. Fergus, Y. LeCun

Key operations in a CNN

Softmax layer:

Source: S. Lazebnik

Loss functions

Gradient descent

Stochastic gradient descent

• Idea: instead of computing the gradient, compute an approximation

​

​

​

​

​

​

Note that in expectation

GD

SGD

How do we optimize the very nonlinear and

nonconvex empirical risk?

​

Many possible choices:

​

• Second-order (Newton-like) methods

​

• Gradient descent

​

• Stochastic gradient descent and variants

Léon Bottou on large-scale learning..

Léon Bottou on large-scale learning..

Léon Bottou on large-scale learning..

Léon Bottou on large-scale learning..

Léon Bottou on large-scale learning..

Léon Bottou on large-scale learning..

Léon Bottou on large-scale learning..

Bottom line:

​

• “Bad” optimization algorithms make sense in the large-scale setting, especially since we minimize expectations

​

• Second-order (Newton-like) methods

​

• Gradient descent

​

• Stochastic gradient descent and variants

Batch SGD