Lecture 7

Neural Networks

6.8300/1 Advances in Computer Vision

Spring 2024

Sara Beery, Kaiming He, Vincent Sitzmann, Mina Konaković Luković

7. Introduction to Deep Learning

• Brief history
• Basic formulation (hierarchical processing)
• Optimization via gradient descent
• Layer types (Linear, Pointwise non-linearity)
• Everything is a tensor
• Deep nets as data transformers

Deep learning

In the past, we didn’t have enough data to fit these models. But now we do!

Modeling the visual world is incredibly complicated. We need high capacity models.

We want a class of high capacity models that are easy to optimize.

Deep neural networks!

A brief history of Neural Networks

Perceptrons, 1958

http://www.ecse.rpi.edu/homepages/nagy/PDF_chrono/2011_Nagy_Pace_FR.pdf. Photo by George Nagy

Rosenblatt

Perceptrons, 1958

Perceptrons,

1958

Minsky and Papert, Perceptrons, 1972

Perceptrons,

1958

Minsky and Papert,

1972

Parallel Distributed Processing (PDP), 1986

XOR problem

Inputs

Output

0 0 0

1 0 1

0 1 1

1 1 0

PDP authors pointed to the backpropagation algorithm

as a breakthrough, allowing multi-layer neural networks to be

trained. Among the functions that a multi-layer network can represent but a single-layer network cannot: the XOR function.

0 1

0 1

Perceptrons,

1958

Minsky and Papert,

1972

PDP book,

1986

LeCun conv nets, 1998

http://yann.lecun.com/exdb/lenet/index.html

Demos:

14

http://pub.clement.farabet.net/ecvw09.pdf

Neural networks to recognize handwritten digits? yes

Neural networks for tougher problems? not really

Neural Information Processing Systems 2000

• Neural Information Processing Systems, is the premier conference on machine learning. Evolved from an interdisciplinary conference to a machine learning conference.
• For the 2000 conference:
• title words predictive of paper acceptance: “Belief Propagation” and “Gaussian”.
• title words predictive of paper rejection: “Neural” and “Network”.

Perceptrons,

1958

Minsky and Papert,

1972

PDP book,

1986

AI winter,

2000

Krizhevsky, Sutskever, and Hinton, NeurIPS 2012

“Alexnet”

Krizhevsky, Sutskever, and Hinton, NeurIPS 2012

Perceptrons,

1958

Minsky and Papert,

1972

PDP book,

1986

AI winter,

2000

Krizhevsky, Sutskever,

Hinton, 2012

What comes next?

Perceptrons,

1958

Minsky and Papert,

1972

PDP book,

1986

AI winter,

2000

Krizhevsky, Sutskever,

Hinton, 2012

2028 ?

What comes next?

Perceptrons,

1958

Minsky and Papert,

1972

PDP book,

1986

AI winter,

2000

Krizhevsky, Sutskever,

Hinton, 2012

2028 ?

[“Mask RCNN”, He et al. 2017]

[“Neural module networks”, Andreas et al. 2017]

[“pix2pix”, Isola et al. 2017]

Image classification

“clown fish”

Edges

Texture

Colors

Segments

Parts

“clown fish”

Classifier

Image classification

“clown fish”

Image classification

“clown fish”

Neural net

Image classification

“clown fish”

Deep neural net

Image classification

“clown fish”

Loss

Deep learning

Gradient descent

Gradient descent

x

Gradient descent

Gradient descent

For large N, computing J in every iteration can be expensive

Stochastic gradient descent (SGD)

• Want to minimize overall loss function J, which is sum of individual losses over each example.
• In Stochastic gradient descent, compute gradient on sub-set (batch) of data.

If batchsize=1 then θ is updated after each example.

If batchsize=N (full set) then this is standard gradient descent.

• Gradient direction is noisy, relative to average over all examples (standard gradient descent).
• Advantages
• Faster: approximate total gradient with small sample
• Implicit regularizer
• Disadvantages
• High variance, unstable updates

Input representation

Output representation

Computation in a neural net

Computation in a neural net

Computation in a neural net

Input representation

Output representation

Linear layer

weights

bias

Computation in a neural net

Input representation

Output representation

Linear layer

weights

bias

parameters of the model

Input representation

Output representation

Computation in a neural net

“Perceptron”

Example: linear classification with a perceptron

One layer neural net (perceptron) can

perform linear classification!

Non-differentiable non-linearity

Input representation

Output representation

Non-linearity with soft activation

Input representation

Output representation

Tanh

Tanh

• Bounded between [-1,+1]
• Saturation for large +/- inputs
• Gradients go to zero (vanishing gradients)
• Outputs centered at 0
• tanh(z) = 2 sigmoid(2z) −1

Computation in a neural net — non-linearity

Sigmoid

• Interpretation as firing rate of neuron
• Bounded between [0,1]
• Saturation for large +/- inputs
• Gradients go to zero
• Outputs centered at 0.5 � (poor conditioning)
• Not used in practice

Computation in a neural net — non-linearity

Rectified linear unit (ReLU)

• Unbounded output (on positive side)
• Efficient to implement:
• Also seems to help convergence (see 6x speedup vs tanh in [Krizhevsky et al.])
• Drawback: if strongly in negative region, unit is dead forever (no gradient).
• Default choice: widely used in current models.

Computation in a neural net — non-linearity

Leaky ReLU

• where α is small (e.g. 0.02)
• Efficient to implement:
• Also known as probabilistic ReLU (PReLU)
• Has non-zero gradients everywhere (unlike ReLU)
• α can also be learned (see Kaiming He et al. 2015).

Computation in a neural net — non-linearity

Input representation

Intermediate representation

Stacking layers - Multi-layer Perceptron (MLP)

Input representation

Intermediate representation

Stacking layers - fully connected layers

Output representation

Input representation

Intermediate representation

Example: how signal evolves

Output representation

Input representation

Intermediate representation

Example: how signal evolves

Output representation

Input representation

Intermediate representation

Example: how signal evolves

Output representation

Input representation

Intermediate representation

Example: how signal evolves

Output representation

Connectivity patterns

Input representation

Output representation

Fully connected layer

Locally connected layer

(Sparse W)

Input representation

Output representation

“clown fish”

Linear

Non-linearity

…

Deep nets

Example: linear classification with a perceptron

One layer neural net (perceptron) can

perform linear classification!

Example: nonlinear classification with a deep net net

• 1 layer? Linear decision surface.
• 2+ layers? In theory, can represent any function. Assuming non-trivial non-linearity.
• Bengio 2009,

http://www.iro.umontreal.ca/~bengioy/papers/ftml.pdf

• Bengio, Courville, Goodfellow book

http://www.deeplearningbook.org/contents/mlp.html

• Simple proof by M. Neilsen

http://neuralnetworksanddeeplearning.com/chap4.html

• D. Mackay book

http://www.inference.phy.cam.ac.uk/mackay/itprnn/ps/482.491.pdf

• But issue is efficiency: very wide two layers vs narrow deep model? In practice, more layers helps.

Representational power

“clown fish”

Linear

Non-linearity

…

Deep nets

Last layer

…

…

dolphin

cat

grizzly bear

angel fish

chameleon

iguana

elephant

clown fish

Classifier layer

“clown fish”

Loss

error

Network output

…

…

dolphin

cat

grizzly bear

angel fish

chameleon

iguana

elephant

clown fish

Ground truth label

Loss function

“clown fish”

Loss

small

Network output

…

…

dolphin

cat

grizzly bear

angel fish

chameleon

iguana

elephant

clown fish

Ground truth label

Loss function

“grizzly bear”

Loss

large

Network output

…

…

dolphin

cat

grizzly bear

angel fish

chameleon

iguana

elephant

clown fish

Ground truth label

Loss function

Network output

…

dolphin

cat

grizzly bear

angel fish

chameleon

clown fish

Ground truth label

…

iguana

elephant

Probability of the observed data under the model

Prediction

dolphin

cat

grizzly bear

angel fish

chameleon

iguana

elephant

clown fish

0

1

Ground truth label

0

1

…

Prediction

dolphin

cat

grizzly bear

angel fish

chameleon

iguana

elephant

clown fish

0

1

Ground truth label

0

1

…

Loss

0

1

Prediction

Ground truth label

dolphin

cat

grizzly bear

angel fish

chameleon

iguana

elephant

clown fish

…

Likelihood

0

1

0

1

0

1

Likelihood of observed true data under predictive model

dolphin

cat

grizzly bear

angel fish

chameleon

iguana

elephant

clown fish

…

0

1

0

1

0

1

Prediction

Ground truth label

Loss

dolphin

cat

grizzly bear

angel fish

chameleon

iguana

elephant

clown fish

…

0

1

0

1

0

1

Prediction

Ground truth label

Loss

Softmax regression (a.k.a. multinomial logistic regression)

“clown fish”

Loss

Deep learning

“grizzly bear”

Loss

Deep learning

“chameleon”

Loss

Deep learning

Batch (parallel) processing

Loss

Loss

Loss

…

Tensors

(multi-dimensional arrays)

…

Furry?

Is a fish?

Size

# Stripes

…

Each layer is a representation of the data

…

Tensors

(multi-dimensional arrays)

# neurons

# features

# units

# “channels”

Everything is a tensor

Regularizing deep nets

Deep nets have millions of parameters!

On many datasets, it is easy to overfit — we may have more free parameters than data points to constrain them.

How can we prevent the network from overfitting?

1. Fewer neurons, fewer layers
2. Weight decay and other regularizers
3. Normalization layers
4. …

Recall: regularized least squares

Regularizing the weights in a neural net

weight decay

“We prefer to keep weights small.”

Dropout

Input representation

Intermediate representation

Output representation

Dropout

Input representation

Intermediate representation

Output representation

Dropout

Input representation

Intermediate representation

Output representation

Dropout

Input representation

Intermediate representation

Output representation

Dropout

Randomly zero out hidden units.

Prevents network from relying too much on spurious correlations between different hidden units.

Can be understood as averaging over an exponential ensemble of subnetworks. This averaging smooths the function, thereby reducing the effective capacity of the network.

Normalization layers

ReLU

Norm

Normalization layers

ReLU

Norm

Normalization layers

ReLU

Norm

Normalization layers

ReLU

Norm

Keep track of mean and variance of a unit (or a population of units) over time.

Standardize unit activations by subtracting mean and dividing by variance.

Squashes units into a standard range, avoiding overflow.

Normalization layers

Also achieves invariance to mean and variance of the training signal.

Both these properties reduce the effective capacity of the model, i.e. regularize the model.

Why do deep nets generalize?

• Deep nets have so many parameters they could just act like look up tables, regurgitating their training data
• Instead, they learn rules that generalize
• Defies classical theory!

The more parameters, the simpler the learned function

[Double-descent: Belkin, Hsu, Ma, Mandal, PNAS 2019]

The simplicity hypothesis

Emerging theory:

deep nets learn simple functions that fit the data

Classical theory:

big models learn complicated functions, and overfit the data

Tensors

(multi-dimensional arrays)

# neurons

# features

# units

# “channels”

“Tensor flow”

z

z

Layer L

Input

Deep nets are data transformers

• Deep nets transform datapoints, layer by layer
• Each layer is a different representation of the data
• We call these representations embeddings

Two different ways to represent a function

Two different ways to represent a function

Data transformations for a variety of neural net layers

Wiring graph

Activations

Parameters

Wiring graph

Equation

Mapping

Matrix

Activations

Parameters

z

logits

class probabilites

relu

softmax

logits

class probabilites

relu

softmax

z

logits

class probabilites

relu

softmax

z

z

Layer 1 representation

[DeCAF, Donahue, Jia, et al. 2013]

[Visualization technique : t-sne, van der Maaten & Hinton, 2008]

7. Introduction to Deep Learning

• Brief history
• Basic formulation (hierarchical processing)
• Optimization via gradient descent
• Layer types (Linear, Pointwise non-linearity)
• Everything is a tensor
• Deep nets as data transformers