1 of 28

Neural Networks for Images

Convolutional Neural Networks — Architectures and other vision tasks

K. P. Murphy, Probabilistic Machine Learning: An Introduction — Chapter 14

2 of 28

2

Contents

14.1

Introduction

14.2

Common Layers

14.3

Architectures

14.4

Other Convolutions

14.5

Other Vision Tasks

3 of 28

3

14.1 Introduction

Recap — what does an MLP hidden layer compute?

Ch. 13: models for unstructured vectors x ∈ ℝᴰ

The value of hidden unit j = inner product of input x with weight vector wⱼ (+ nonlinearity φ)

We can read wⱼ as a learned 'template' or pattern.

Large inner product = the input matches the template well

With ReLU, a good match yields a large activation → a signal that "pattern j is present in the input".

So MLP layer = template matching

“Inner product = similarity measurement” — a CNN applies this inner product 'small, repeatedly, at every location'.

4 of 28

4

14.1 Introduction

Three reasons plain MLPs fail on images

problems that appear the moment the input is x ∈ ℝ^(W×H×C)

① Variable-sized input

If the image size changes, the weight matrix W must change too — a different model per size?

② Parameter

W has size (W×H×C)×D.

E.g. 224×224×3 input, 1024 hidden units

→ ~150 million parameters (one layer!)

③ No translation invariance

Weights are not shared across locations — a pattern learned on the left is not recognized on the right

Common root cause: the model ignores the 2D spatial structure (locality, repeated statistics) of images

→ Solution: replace matrix multiplication with convolution

5 of 28

5

14.1 Introduction

The CNN prescription — share a small filter across all locations

Murphy, PML Fig. 14.2 — classify a digit by discriminative part-templates in the right (relative) locations

Split the image into overlapping 2D patches; compare each patch with small filters (3×3, 5×5)

A filter = a learned template for a 'part' of an object. Learned from data.

Small templates → far fewer parameters

A 3×3×3 filter on a 224×224 input = just 27 parameters (independent of position).

The same filter at every location → robust to translation

The same response wherever the object sits — exactly what classification needs.

Variable-sized inputs are fine too

Filter size is independent of input size — only the output size changes.

6 of 28

8

14.2 Common Layers

1D discrete convolution and cross-correlation

Equation of 1d convolution:

Murphy, PML Fig. 14.3 — example of 1d discrete convolution

(Equation of 1d cross-correlation, Eq. 14.4)

Murphy, PML Fig. 14.4 — 1d cross-correlation

Each output zᵢ = multiply and sum with flipped w

Don't flip w → cross-correlation (Eq. 14.3)

Identical when w is symmetric. In deep learning w is learned anyway, so the flip is irrelevant → by convention, cross-correlation is simply called convolution.

Note the name: cross-“correlation”

Same root as statistical correlation — at each position it measures 'how much x and w are similar'.

7 of 28

7

14.2 Common Layers

Why this operation?

Intuition ① — the distribution of a sum of two random variables IS a convolution

Sum over all pairs (u, z−u) whose total is z, multiplying probabilities — the g(z−u) in the definition is exactly this structure.

Intuition ② — Each output is inner product

Cross correlation can be interpreted as similarity score between x and w

8 of 28

9

14.2 Common Layers

Each output is 'filter · patch inner product' = a similarity score

Geometric meaning: inner product ∝ cos θ — larger when the patch 'points the same way' as the filter

Filter = template (direction), output = magnitude of that direction's component. A matched filter is the optimal linear detector of a signal in white noise.

Statistical meaning: (zero-mean) inner product ∝ sample covariance/correlation of filter and patch

The output map = a map of "how correlated is the local pixel arrangement here with the filter pattern?"

Probabilistic meaning: kernel = local weight distribution → output = local expectation/contrast

All-positive kernel = local mean (low-pass); mixed-sign kernel = local difference (high-pass, edges).

Cross-correlation spikes

when the pattern is present

(the matched-filter effect)

9 of 28

10

14.2 Common Layers

2D convolution

slide the filter W (H×W) over the image X in 2D (Eq. 14.5)

Murphy, PML Fig. 14.5 — example of 2d convolution

Murphy, PML Fig. 14.6 — response map of a diagonal-edge filter

Each output pixel = the inner product of the 2D patch at that position with the kernel

Slide the window one step at a time: 3×3 input × 2×2 kernel → 2×2 output.

Template matching: the map 'lights up’ where the filter responds → a heat map

We can think of convolution as a form of feature detection. This is why Y = W⊛X is called a feature map.

10 of 28

11

14.2 Common Layers

Synthesis — why can kernel filters extract information from images?

three viewpoints, one conclusion

Signal processing

Matched filter: a filter shaped like the target pattern is the optimal linear detector (in white noise).

Output = an evidence map for the pattern's presence.

Statistics

Output ∝ correlation/covariance between filter and patch.

Natural-image statistics are stationary in position → sharing one filter across all locations is statistically justified.

Frequency

Convolution theorem: convolution in space = multiplication in frequency.

Filter = a selector of frequency/orientation components (edges = high-freq, blur = low-freq).

One layer = linear similarity measurement + nonlinearity (ReLU) — stacking layers builds a hierarchy of 'patterns of patterns' (edge → texture → part → object)
A single pixel is meaningless, but the 'arrangement' of a local patch carries information — convolution is precisely the operation that reads that arrangement

11 of 28

12

14.2 Common Layers

Convolution = matrix multiplication with special structure

Murphy, PML Eq. 14.8–14.9 — Toeplitz-like matrix C

Flatten X into a vector, build a matrix C from the kernel: y = Cx

Convolution is linear — a special case of matrix multiplication.

The structure of C is the essence of a CNN

① Mostly zeros (sparse weight matrix)

② The same values w₁…w₄ repeat along diagonals (weight tying = sharing across positions)

The result

Translation equivariance + a dramatic reduction in parameters.

12 of 28

13

14.2 Common Layers

Padding and stride

Murphy, PML Fig. 14.8 — (a) zero-padding same conv: 5×7 preserved (b) stride 2 → 3×4

Same convolution: output=input size

Zero-padding adds a border of 0s; with 2p = f−1 you get same convolution.

p=padding size, f=filter size

stride s: neighboring outputs share receptive fields, so values are similar → skip every s-th position

In general, output has the following size:

13 of 28

14

14.2 Common Layers

Multiple input & output channels

C input channels → kernel W ∈ ℝ^(H×W×C×D) → D output channels (filters)

Murphy, PML Fig. 14.9 — 2-channel input: per-channel conv, then sum

Murphy, PML Fig. 14.10 — cylinders = hypercolumns

Input channels: one 2D filter per channel, results summed — output pixel = inner product of the 'spatial patch × all channels' volume with the kernel volume (still an inner product!)

Output channels: D filters = D parallel 'similarity questions' → hypercolumn z(i,j,1:D) = the D-dim feature vector at one location

Stacking layers: the next layer template-matches the 'spatial arrangement of features' → the edge → texture → part → object hierarchy.

14 of 28

15

14.2 Common Layers

1×1 (pointwise) convolution and pooling

Murphy, PML Fig. 14.11 — 1×1×3×2: 3 channels → 2

Murphy, PML Fig. 14.12 — 2×2 max pooling

1×1 conv: the same small MLP applied in parallel to each location's hypercolumn

Uses: channel reduction/expansion (bottleneck), feature recombination

Pooling: the case we want local invariance

Keep only the max(or average, mean, etc…) in the window → output unchanged under small shifts.

Global average pooling (GAP): H×W×D → 1×1×D

For classification, we can apply classifier to an any size of image, because GAP will always converts feature map to fixed D-dimension feature.

15 of 28

16

14.2 Common Layers

Assembling the parts — the classic CNN design pattern

[Conv + ReLU → Pool] repeated → Flatten → FC → Softmax

Murphy, PML Fig. 14.13 — feature-learning stage + classification stage

16 of 28

17

14.2 Common Layers

Normalization layers-proposed to mitigate internal covariate shift

Batch Normalization

BN: standardize with minibatch statistics (μ_B, σ²_B), then restore with learnable γ, β

Internal layer distributions keep shifting during training (internal covariate shift) → recompute per batch.

At test time: 'freeze' μ, σ² computed on the full data to cover single batch

Murphy, PML Fig. 14.14 — blue region = normalized by the different (μ,σ)

BN statistics are unreliable with small batches → pool other axes: Layer (channel·space) / Instance (space only) / Group (channel groups) Norm
Difference between BN with B=1 and Instance: In test phase, BN use frozen mean and variance.
Filter Response Normalization(with Threshold Linear Unit): divide by mean square norm rather than normalization, TLU covers mean centering.

Replace activation vector, or pre-activation vector

17 of 28

18

14.3 Architectures

LeNet (1998) — the prototype of the modern CNN

Murphy, PML Fig. 14.15 — LeNet5

Murphy, PML Fig. 14.17 — predictions after 1 epoch (red = wrong)

Structure: [Conv 5×5 → AvgPool] ×2 → FC(120) → FC(84) → FC(10)

Alternating conv-pool + final FC classifier — exactly the design pattern from 14.2.

Performance: 98.8% on MNIST after just 1 epoch (the comparable MLP: 95.9%)

A quantitative demonstration of the gains from convolution's inductive bias.

Significance: one of the first practical CNNs trained end-to-end with backprop + SGD

18 of 28

19

14.3 Architectures

AlexNet (2012) — the trigger of the deep-learning boom

Murphy, PML Fig. 14.16 — (a) LeNet5 vs (b) AlexNet

'Almost the same' design as LeNet, but:

① deeper (5 → 8 trainable layers) ② tanh → ReLU ③ weight decay → dropout ④ conv layers stacked back-to-back (no strict conv-pool alternation)

Why stack 3×3s?

Three 3×3s = a 7×7 receptive field, but fewer parameters

Scale: 60M parameters (more than the 1M labels!)

Mostly in the 3 final FC layers. Trained split across 2 GPUs

The idea (CNNs) existed in 1998 — what was missing was data (ImageNet), compute (GPUs), and a few training tricks

19 of 28

20

14.3 Architectures

GoogLeNet / Inception (2014) — don't choose a filter size, use them all

Murphy, PML Fig. 14.18 — Inception module

Murphy, PML Fig. 14.19 — full model (9 inception blocks + GAP)

Inception block: parallel 1×1 / 3×3 / 5×5 / pool paths → concat along channels

The model itself learns, per layer, which scale of feature is useful.

1×1 convs shrink channels at the entrance of each path

The bottleneck controls the compute cost of going parallel

Global average pooling instead of final FC layers

Removes AlexNet's FC parameter bomb → a much lighter model.

20 of 28

21

14.3 Architectures

ResNet (2015) — introducing residual block

Murphy, PML Fig. 14.20 — residual block (right: 1×1 conv to match channels)

Skip connections = gradient highways

Gradients flow directly from the output to early layers → very deep training becomes possible.

PreResNet : add input after calculating φ

Murphy, PML Fig. 14.21 — ResNet-18: (Conv:BN:Max):(R:R):(R′:R)×3:Avg:FC — input reduced 2⁵=32× before GAP

21 of 28

22

14.3 Architectures

DenseNet (2017) — concatenate, don't add

every layer receives the outputs of ALL previous layers directly

Murphy, PML Fig. 14.22 — (a) add vs concat, (b) dense connectivity

Replace ResNet's add with concat

Instead of add earlier features, concatenate them so later layers can use them directly.

Pros: feature reuse + direct gradients → reported to outperform ResNets

All previously computed features are directly accessible to the output layer.

Cons: channels stack up with depth → more parameters, memory, compute

Mitigated by interleaved 1×1 conv layers and stride-2 pooling.

22 of 28

23

14.3 Architectures

Neural Architecture Search — learning the design itself

the networks so far = different arrangements of the same blocks → search that arrangement automatically

Observation: CNN design is repeated block rearrangement + hyperparameter tuning

ConvNeXt (2022) is a SOTA CNN built by stacking small refinements on a standard ResNet

NAS = black-box optimization of architectures to minimize validation loss (a branch of AutoML)

Can optimize multiple objectives — accuracy plus model size and inference speed — this is how EfficientNetv2 was created.

23 of 28

24

14.4 Other Convolutions

Transposed convolution — from a small input to a larger output

'stencil stamping': the kernel is the stamp, the input value is the ink intensity

Murphy, PML Fig. 14.24 — 2×2 input × 2×2 kernel → 3×3 output

For each input position: 'stamp' a copy of the kernel onto the output canvas, weighted by that pixel value; sum where copies overlap
Use case: upsampling — segmentation decoders, image generation, anywhere resolution must be 'restored'
Origin of the name: writing conv as a matrix C, this operation is exactly multiplication by Cᵀ — a transpose, not an inverse

Murphy, PML Fig. 14.25 — terminology warning: NOT deconvolution (the inverse problem of undoing a known filter)!

24 of 28

25

14.4 Other Convolutions

Depthwise separable convolution — separate space from channels

split standard conv (H×W×C×D) into two steps to cut cost (the core of MobileNet & edge models)

Murphy, PML Fig. 14.26 — per-channel 2D conv → 1×1 conv combines channels

Step 1 (depthwise): each channel gets its own 2D filter — looks at space only

Step 2 (pointwise 1×1): mix channels C → D — looks at channels only

Example: 12×12×3 input, 5×5 filter, 256 output channels

Standard: 5·5·3·256 = 19,200 parameters

Separable: 5·5·3 + 1·1·3·256 = 843 parameters (≈23× fewer; output still 8×8×256)

25 of 28

26

14.5 Other Vision Tasks

Image tagging — multiple labels per image

classification (mutually exclusive single label) → multi-label prediction

Output space Y = {0,1}^C — an independent yes/no per tag(C is number of tag types)

Just replace the final softmax with C independent logistic (sigmoid) units.

Provide a free way to generate large datasets: social-media hashtags

Noisy (many tags aren't visually well defined), but useful for pre-training.

Arguably a MORE sensible objective than classification

Real images contain multiple objects — which one should be 'the' label? Karpathy's human-benchmark analysis on ImageNet: 16% of human errors come from 'multiple objects but only one correct label'.

26 of 28

27

14.5 Other Vision Tasks

Object detection — what, where, and however many

variable-count outputs: closing an open world with anchor boxes

Murphy, PML Fig. 14.27 — (a) face detection (b) anchor boxes

Anchor boxes: a pre-laid grid of candidate positions, scales, aspect ratios

For each box, predict ① object presence probability ② class ③ box offsets (position refinement).

Representative models: SSD, YOLO (single-shot family)

Many variants trading off speed, accuracy, simplicity.

Face detection = the special case with one class

(distinct from face recognition — identifying who it is)

27 of 28

30

14.5 Other Vision Tasks

Human pose estimation — finding skeletal keypoints

predict the locations of a fixed set of keypoints: head, hands, joints …

Murphy, PML Fig. 14.32 — OpenPose: body, hand, and face keypoints

Another extension of detection: skeletal keypoint sets instead of boxes

Representative systems: PersonLab, OpenPose — real-time, multi-person.

The same template extends to 3D

The hard part is 3D labels — humans struggle to annotate in 3D → use graphics engines to synthesize data with unlimited ground truth.

Applications

Action recognition, AR/VR, sports analytics, human-robot interaction.

28 of 28

31

Summary

1

Each convolution output = the inner product of the filter with a local patch = a similarity (correlation) score

Why kernel filters extract image information

2

A CNN = a sparse, tied MLP with locality and weight sharing hard-coded

3

The history of architectures = the history of improving gradient flow and parameter efficiency

LeNet → AlexNet (depth·ReLU) → Inception (parallel) → ResNet (residual) → DenseNet (connectivity) → NAS (automation).

References: K. P. Murphy, Probabilistic Machine Learning: An Introduction, Ch. 14 (probml.github.io) · 3Blue1Brown, “But what is a convolution?” (youtube.com/watch?v=KuXjwB4LzSA)