Convolutional neural networks
Slides from Fei-Fei Li, Justin Johnson, Serena Yeung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
(Cornell University)
Hinton and Salakhutdinov. Reducing the Dimensionality of Data with Neural Networks. Science, 2016.
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Fast-forward to today: ConvNets* are everywhere
* and other recent architectures, like Transformers
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Fast-forward to today: ConvNets are everywhere
Self-driving cars (video courtesy Tesla)
Cloud TPU v4 Pods
Text-to-image
“A computer vision class watching a cool lecture, crayon drawing”
“A computer vision class watching a cool lecture, album cover”
What is a ConvNet?
Motivation – Feature Learning
Life Before Deep Learning
Input Pixels
Extract Hand-Crafted Features
Figure: Karpathy 2016
Concatenate into a vector x
Linear Classifier
Ans
SVM
Why use features? Why not pixels?
Slide from Karpathy 2016
Q: What would be a
very hard set of classes for a linear classifier to distinguish?
(assuming x = pixels)
Goal: linearly separable classes
Aside: Image Features
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Image Features: Motivation
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Image Features: Motivation
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Last layer of many CNNs is a linear classifier
Input Pixels
Ans
Perform everything with a big neural network, trained end-to-end
This piece is just a linear classifier
Key: perform enough processing so that by the time you get to the end of the network, the classes are linearly separable
(GoogLeNet)
Visualizing AlexNet in 2D with t-SNE
[Donahue, “DeCAF: DeCAF: A Deep Convolutional …”, arXiv 2013]
(2D visualization using t-SNE)
Linear Classifier
Convolutional neural networks
Layer types:
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Number of weights: 5 x 5 x 3 + 1 = 76
(vs. 3072 for a fully-connected layer)
(+1 for bias term)
Adapted from Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
(total number of parameters to learn: 6 x (75 + 1) = 456)
How many parameters are in a convolution layer consisting of 3 3x3x1 filters (each with bias term)?
ⓘ Start presenting to display the poll results on this slide.
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
CNNs: from image pixels to learned feature hierarchies
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
68
Local connectivity
Each output unit combines a small spatial neighborhood rather than every input pixel. This builds local patterns such as edges, corners and textures.
Shared weights
The same kernel is reused at every spatial location. A feature detector can therefore respond wherever the pattern occurs.
Feature maps / channels
Each learned kernel produces one output channel. Multiple channels represent different learned responses in parallel.
A CNN is a deep network whose central operation is a trainable, multi-channel convolution.
Early layers tend to capture local visual structure; deeper layers combine these responses into more discriminative representations.
Compared with dense connections, local connectivity and weight sharing greatly reduce the number of learnable parameters.
A convolutional layer: multiple input and output channels
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
69
Input tensor
H × W spatial grid with Cᵢₙ channels. For RGB input, Cᵢₙ = 3; deeper layers may have many more channels.
Learned filters
Each output channel has its own kernel bank spanning all input channels: Kₕ × Kᵥ × Cᵢₙ weights, plus one bias.
Output tensor
Cₒᵤₜ feature maps. Each output pixel is a weighted sum over the local spatial window and all input channels, followed by an activation.
s(i,j,cₒ) = Σcᵢ Σ(Δx,Δy) w(Δx,Δy,cᵢ,cₒ) x(i+Δx,j+Δy,cᵢ) + b(cₒ)
The same set of weights is applied at every valid spatial location. In common deep-learning libraries this operation is often called cross-correlation; the learned weights make the kernel reversal convention immaterial.
Parameter sharing: why convolution is economical
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
70
Parameters = Kₕ × Kᵥ × Cᵢₙ × Cₒᵤₜ + Cₒᵤₜ
Including one bias per output channel
Dense layer: H × W × Cᵢₙ × H′ × W′ × Cₒᵤₜ
Illustrative dense spatial connectivity
Worked example
3 × 3 kernel, 3 input channels, 6 output channels:�3 × 3 × 3 × 6 + 6 = 168 parameters.
Important consequence
Kernel-weight gradients are accumulated over all spatial positions where that kernel is used. The learned detector is shared across the image.
The book’s Figure 5.34 illustrates 6 input channels, 4 output channels and 3 × 3 kernels: 9 × 6 × 4 = 216 kernel weights (biases excluded).
Convolution geometry: output size, padding and stride
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
71
Hₒᵤₜ = ⌊(H + 2P − D(K−1) − 1)/S⌋ + 1
Same expression applies to width
Padding (P)
Adds border pixels (often zeros) so edge locations can be processed and spatial dimensions can be preserved when configured appropriately.
Stride (S)
Moves the kernel by more than one pixel. Larger stride reduces output resolution; AlexNet’s first convolution used stride 4.
Dilation (D)
Spaces kernel samples apart. It expands the receptive field without proportionally increasing kernel parameters or dense multiply-adds.
Example: H = 32, K = 3, P = 1, S = 1, D = 1 → Hₒᵤₜ = 32. With S = 2, the output is downsampled to 16.
Grouped and depthwise convolution
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
72
Standard convolution
Every output channel can use all input channels. Parameter count (without bias): KₕKᵥCᵢₙCₒᵤₜ.
Grouped convolution
Split channels into G groups; each group is convolved separately. This reduces cross-channel connectivity and computation.
Depthwise convolution
Special case G = Cᵢₙ: each input channel is convolved independently. Often followed by 1 × 1 pointwise convolution to mix channels.
Why it matters: efficient CNNs use channel grouping to reduce model size and computation.
Szeliski connects depthwise / channel-separated convolution to MobileNets and discusses grouped convolution in ResNeXt. These are architectural tools for balancing accuracy, model size and runtime.
Pooling and downsampling
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
73
Max pooling
For each local window, retain the maximum activation. Common choices: 2 × 2 window with stride 2, or overlapping 3 × 3 windows.
Average pooling
Compute the mean response in a local window, typically channel-wise. Can be viewed as a fixed averaging kernel plus downsampling.
Strided convolution
A learned alternative to fixed pooling: convolution evaluated at spaced positions can reduce resolution while learning the downsampling filters.
Backpropagation through max pooling
The forward pass records which input location supplied the maximum. During backpropagation, the gradient is routed to that selected location; other positions in the window receive zero from that pooling output.
Caution: pooling is often described as providing shift invariance, but CNNs are not automatically highly shift-invariant.
Unpooling and transposed convolution
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
74
Max unpooling
Use the stored max-pooling switches to place activations back at their selected locations. This approximately reverses the spatial reduction, leaving other positions empty.
Transposed convolution
Insert (S−1) zeros between input samples, then apply a learned convolution. The stride controls the upsampling factor.
Downsample → learned / fixed feature processing → upsample
Transposed convolution is the transpose of the sparse matrix representation of a convolution operator; it is not generally the exact inverse of convolution. Padding, stride and output padding determine the resulting spatial dimensions.
Encoder–decoder CNNs: fully convolutional networks and U-Net
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
75
Encoder
Repeated convolution blocks; spatial resolution decreases while channel count typically increases.
Bottleneck
Compact, low-resolution representation with a large receptive field.
Decoder
Upsampling and convolution progressively restore spatial resolution.
Dense output
Per-pixel labels, image restoration, depth or other spatial predictions.
U-Net adds skip connections between encoder and decoder stages at matching resolutions.
These lateral links preserve fine spatial detail that may be lost during downsampling, while the bottleneck captures broader context. The book also relates this design to feature-pyramid networks and other multi-resolution vision tasks.
Applications: semantic segmentation, image denoising, super-resolution and monocular depth estimation.
LeNet-5: a compact CNN for digit recognition
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
76
Architecture pattern
Input image → multi-channel convolution → downsampling → convolution → downsampling → fully connected layers → 10 outputs.
Historical design
LeNet-5 used sigmoid nonlinearities, average pooling and Gaussian RBF output units rather than the ReLU / max-pooling / softmax combination common in later networks.
Practical impact
Originally deployed for automatic check reading. The book notes that variants of this architecture remain useful teaching examples.
Key lesson: a CNN alternates feature extraction and spatial reduction before classification.
MNIST is a classic introductory dataset; CIFAR-10 and Fashion-MNIST provide more challenging image-classification exercises.
From AlexNet to ResNet: major architectural ideas
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
77
AlexNet (2012)
ReLU, max pooling, convolutional feature hierarchy, dropout, data augmentation; major ImageNet error reduction.
VGG (2014)
Repeated 3 × 3 convolution–ReLU blocks, 2 × 2 max pooling and channel doubling; strong accuracy, higher computation.
GoogLeNet / Inception (2015)
Parallel branches with multiple filter sizes; 1 × 1 bottlenecks reduce channel dimensionality; global average pooling.
ResNet (2015)
Skip / residual connections improve information and gradient flow, enabling much deeper networks (e.g., 18–152 layers).
Deeper networks improved accuracy, but often increased computational cost.
Efficient CNNs: accuracy, parameters and computation
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
78
Model size
Usually measured by the number of learned parameters (weights and biases); affects storage and memory requirements.
Compute
Often measured in FLOPs / operations per inference pass; affects latency and energy use.
Mobile networks
MobileNet uses depthwise convolutions; width and resolution multipliers tune the accuracy–size–efficiency trade-off.
Other efficiency ideas covered by Szeliski
Grouped convolutions (e.g., ResNeXt) and channel shuffling (ShuffleNet).
Lower-precision arithmetic and weight compression, including binary-weight approaches.
Neural Architecture Search (NAS) to explore network topologies and parameter settings.
Model zoos, transfer learning and fine-tuning
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
79
Start from a pretrained model
A model zoo provides established architectures and pretrained weights (e.g., TorchVision / timm).
Replace the head
Keep the backbone, replace the final classifier or task-specific head with one suited to the target labels.
Fine-tune carefully
Train the new head; optionally unfreeze some backbone layers using a smaller learning rate and sufficient target data.
Transfer learning = pretraining on one dataset / task and adapting the learned representation to another.
Practical checklist: align preprocessing and input size; check class imbalance; freeze/unfreeze deliberately; validate on target-domain data; compare against a from-scratch baseline when feasible.
Visualizing CNN weights, activations and decisions
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
80
Filter inspection
First-layer kernels can often be visualized directly. They commonly reveal simple oriented or color-sensitive patterns.
Activation inspection
Find input patches that maximally activate a channel; deeper channels respond to larger, more complex image regions.
Input attribution
Occlusion maps, saliency maps and Grad-CAM indicate which image regions influence a channel or class response.
Why visualize?
Visualization helps develop intuition, debug failure modes and assess whether a network is using plausible image evidence. It does not, by itself, prove that a model’s reasoning is correct.
CNN robustness: adversarial examples and dataset bias
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
81
Adversarial examples
Small, deliberately chosen perturbations can change a network’s prediction while leaving the image visually similar to a person. White-box attacks use access to model weights / gradients.
Why this matters
Models may exploit non-robust or spurious correlations in training data. High test accuracy on one dataset does not guarantee robust behavior under distribution shift.
Evaluation and mitigation
Test on varied data sources and cross-dataset settings; inspect failure cases.
Use adversarial training / robustness evaluation when the application requires it.
Treat saliency and visualization as diagnostic aids, not guarantees of safety.
Self-supervised pretraining: reusable CNN representations
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
82
Pretrain without labels
Learn useful image representations from large collections of unlabeled images using a proxy / pretext objective.
Adapt to the task
Replace or extend the prediction head and fine-tune on a smaller labeled dataset for classification or another vision task.
Why useful
Can reduce dependence on large manually labeled datasets and provide a strong initialization for downstream tasks.
Szeliski’s §5.4.7 discusses self-supervised learning in the context of pretraining a backbone and adapting it to new tasks. This slide is a conceptual bridge; the original 67-slide deck does not cover this topic in detail.
Takeaway: learned visual features can be reused, but target-domain validation remains essential.
CNN design workflow: from problem to validated model
Source: Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., §5.4 (pp. 291–312)
83
1 Define task
Image classification, detection, segmentation or restoration.
2 Prepare data
Representative train / validation / test splits; labels, preprocessing and augmentation.
3 Choose backbone
Start with a suitable model; consider accuracy, parameter count, FLOPs and deployment limits.
4 Train / fine-tune
Select loss, optimizer, learning-rate schedule and regularization; monitor validation performance.
5 Inspect & test
Visualize activations and errors; evaluate robustness and cross-domain generalization.
Book connection: the chapter emphasizes model structure, training, visualization, computational cost and failure modes.
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
“1x1 convolutions”
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Convolutional layer—properties
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
Slide credit: Fei-Fei Li & Andrej Karpathy & Serena Leung
AlexNet (2012)
Output: 1,000-D vector (probabilities over 1,000 ImageNet categories)
Elgendy, Deep Learning for Vision Systems, https://livebook.manning.com/book/grokking-deep-learning-for-computer-vision/chapter-5/v-3/
6M parameters in total
Big picture
Data is key—enter ImageNet
Performance improvements on ILSVRC
AlexNet
Pre-deep learning era
{
Deep learning era
Image credit: Zaid Alyafeai, Lahouari Ghouti