Image Classification

Victor Boutin

Outline

• Motivation
• Framework for classification
• Overview
• Challenges
• Pipeline
• Receptive field and convolutions
• Datasets
• Models of classification
• Pre alexnet
• Post alexnet
• Trends in deep learning (if time permits)
• Practical session

Introduction

Overview of recognition tasks

https://arxiv.org/pdf/1405.0312.pdf

What is Computer Vision (CV)?

5

What is Computer Vision (CV)?

6

Task: Classification

7

• goldfish
• pigeon

…

• skateboard
• bike
• rugby ball

…

• sandwich
• egg

…

Potential Categories:

Task: Classification

8

• goldfish
• pigeon

…

• skateboard
• bike
• rugby ball

…

• sandwich
• egg

…

Potential Categories:

Task: Object Detection

9

Task: Object Detection

10

Task: Image Segmentation

11

Task: Image Segmentation

12

Automatic

Tumor

segmentation

Image Classification

Image Classification

​

Image Classification

​

dog

car

cat

…

Set of categories

Image Classification

​

dog

car

cat

…

✅

❌

❌

Set of categories

Problem

Problem

Problem

168 145 117

Each pixel:

RGB ([R]ed [G]reen [B]lue) representation

Problem

RGB Matrices

Image Classification

​

dog

car

cat

…

Set of categories

Image Classification

​

dog

car

cat

…

Set of categories

82 %

9 %

0.5 %

8.5 %

Challenges

• Viewpoint variation. A single instance of an object can be oriented in many ways with respect to the camera.
• Scale variation. Visual classes often exhibit variation in their size (size in the real world, not only in terms of their extent in the image).
• Deformation. Many objects of interest are not rigid bodies and can be deformed in extreme ways.
• Occlusion. The objects of interest can be occluded. Sometimes only a small portion of an object (as little as few pixels) could be visible.
• Illumination conditions. The effects of illumination are drastic on the pixel level.
• Background clutter. The objects of interest may blend into their environment, making them hard to identify.
• Intra-class variation. The classes of interest can often be relatively broad, such as chair. There are many different types of these objects, each with their own appearance,

https://cs231n.github.io/classification/

​

Challenges

Framework for classification

What matters

Machine learning methods (e.g., linear classification, deep learning)

Representation (e.g., SIFT, HoG, deep learned features)

Data (e.g., PASCAL, ImageNet, COCO)

The statistical learning framework

Apply a prediction function to a feature representation of the image to get the desired output:

​

f

The statistical learning framework

Apply a prediction function to a feature representation of the image to get the desired output:

​

f

Feature representation

x

Output

y

The statistical learning framework

Apply a prediction function to a feature representation of the image to get the desired output:

​

The statistical learning framework

Apply a prediction function to a feature representation of the image to get the desired output:

​

X

y

The statistical learning framework

• Training: given a training set of labeled examples {(x₁,y₁), …, (x_N,y_N)}, estimate the prediction function f by minimizing the prediction error on the training set

• Testing: apply f to a never before seen test example x and output the predicted value f(x) = y

Image categorization

An image classifier:

Data-driven approach

• Collect a database of images with labels

• Use ML to train an image classifier

• Evaluate the classifier on test images

​

​

Data-driven approach

What we need:

• A database of images with labels
• Use ML to train an image classifier
• Evaluate the classifier on test images

​

​

Linear Classifier

Scores

dog

car

cat

…

82 %

9 %

0.5 %

8.5 %

Scores

dog

car

cat

…

12.3

2.2

-5.3

→ softmax

f

Probabilities from scores?

Solution:

Objective:

• 0 ≤ p ≤ 1
• p_dog + p_cat + p_car + ... = 1

Scores:

• “y”s can have any value
• Does not sum to 1

Probabilities from scores?

Objective:

• 0 ≤ p ≤ 1
• p_dog + p_cat + p_car + ... = 1

Scores:

• “y”s can have any value
• Does not sum to 1

Solution:

• S = y_dog + y_cat + y_car + …
• p_dog = y_dog / S
• p_dog + p_cat + p_car + … = 1

BUT problem:

p_dog can be negative and S can be 0!

Solution: Use exp (strictly increasing and strictly positive).

​

Softmax

Score function

Class score

How to learn f?

dog

car

cat

…

12.3

2.2

-5.3

f(x)

What is learning?

f₁

f₂

f₃

f₄

Set of functions

f^*

Best function

Goal:

Find the function that optimizes a criterion.

​

Criterion:

Best classification scores across a hold-out test set.

Parameterization

f₁

f₂

f₃

f₄

Set of functions

f^*

Best function

f(x, W)

W₁

W₂

W₃

W₄

W^*

Set of parameters

How to learn f?

dog

car

cat

…

12.3

2.2

-5.3

f(x, W)

Score function: f

Parametric approach

​

How to learn f?

dog

car

cat

…

12.3

2.2

-5.3

f(x, W)

f: a neural Network

W: the weights

MLP

Parametric approach

​

MLP

Parametric approach

​

vector

tensor

MLP

Parametric approach

​

vector

= 3072

Vectorize the image

Score function: f

Parametric approach

​

= 3072

Score function: f

Parametric approach

​

= 3072

bias

Linear Classifier

Linear Classifier

Linear Classifier

Interpretation

Template matching

f(x_i, W, b) = Wx_i + b

Geometric Interpretation

Linear classifier: Three viewpoints

Convolution & Receptive field

Convolution Layer

Convolution Layer

​

Filters always extend the full depth of the input volume

Sliding window

​

3x3 window sliding over the image

Source: https://www.kaggle.com/code/ryanholbrook/the-sliding-window

Input image

output

Sliding kernel

Sliding window: stride

​

Shift of the kernel: here we shift the window by 2 pixels in each direction:

stride = (2, 2)

Source: https://www.kaggle.com/code/ryanholbrook/the-sliding-window

Input image

output

Sliding kernel

Sliding window: padding

​

Adding extra values on the border of the image.

Useful to have an output size = input size.

Source: https://www.kaggle.com/code/ryanholbrook/the-sliding-window

Input image

output

Sliding kernel

Convolution Layer

​

https://setosa.io/ev/image-kernels/

​

Convolution Layer

​

Convolution Layer

​

consider a second, green filter

Convolution Layer

​

https://miro.medium.com/max/4800/1*QgiVWSD6GscHh9nt55EfXg.gif

Pooling layer

• makes the representations smaller and more manageable
• operates over each activation map independently

​

Max Pooling

Receptive field

The receptive field is the area of the input image that is seen by single neurons in any layer of the network

​

Receptive field

The receptive field is the area of the input image that is seen by single neurons in any layer of the network

​

Receptive field

The receptive field is the area of the input image that is seen by single neurons in any layer of the network

​

Receptive field

The receptive field is the area of the input image that is seen by single neurons in any layer of the network

​

Receptive field

The receptive field is the area of the input image that is seen by single neurons in any layer of the network

​

Receptive field

The receptive field is the area of the input image that is seen by single neurons in any layer of the network

​

Receptive field: Convolutions

The receptive field is the area of the input image that is seen by single neurons in any layer of the network

Example: 2 convolutional layers with 3x3 filters

​

Receptive field

Example: 2 convolutional layers with 3x3 filters

Each activation map #1 “sees” an area of 3x3 in the input

​

​

Receptive field

Example: 2 convolutional layers with 3x3 filters

Each activation map #2 “sees” an area of 3x3 in map #1

​

​

Receptive field

​

• How much of the input does an activation of map #2 see?

​

Receptive field

​

How much of the input does an activation of map #2 see?

​

It sees a 5x5 region! WHY?

​

Receptive field

​

How much of the input does an activation of map #2 see?

​

It sees a 5x5 region! WHY?

​

​

Receptive field

​

How much of the input does an activation of map #2 see?

​

It sees a 5x5 region! WHY?

​

​

​

Receptive field

​

How much of the input does an activation of map #2 see?

​

It sees a 5x5 region! WHY?

​

​

​

Receptive field

​

How much of the input does an activation of map #2 see?

​

It sees a 5x5 region! WHY?

​

​

​

Receptive field

​

How much of the input does an activation of map #2 see?

​

It sees a 5x5 region! WHY?

​

​

​

Receptive field

​

How much of the input does an activation of map #2 see?

​

It sees a 5x5 region! WHY?

​

​

​

Receptive field

​

How much of the input does an activation of map #2 see?

​

It sees a 5x5 region! WHY?

​

​

​

Receptive field

​

How much of the input does an activation of map #2 see?

​

It sees a 5x5 region! WHY?

​

​

​

Receptive field

​

How much of the input does an activation of map #2 see?

​

It sees a 5x5 region! WHY?

​

​

​

Receptive field

​

Receptive field

​

bias

Receptive field

​

Factors that affect the receptive field:

• Number of layers

• Filter size

​

26

Two stacked layers of a 3x3 filter give the same receptive field as one layer of a 5x5 filter

Receptive field

​

Factors that affect the receptive field:

• Number of layers

• Filter size

​

Presence of max-pooling

• Increase receptive field

• Decrease resolution

• Spatial information is lost

​

26

Two stacked layers of a 3x3 filter give the same receptive field as one layer of a 5x5 filter

Visual Transformers (ViTs)

Transformers

Details in a future lesson…

​

x₁

x₂

x₃

x_n

Transformer

…

Input: sequence

Image as a sequence

?

Image as a sequence

Transformers

Transformer

Datasets

Datasets

https://paperswithcode.com/sota

​

Datasets

• MNIST
• CIFAR-10
• CIFAR-100
• ILSVRC2012

https://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html#494c5356524332303132207461736b2031

MNIST

Classic dataset of handwritten images for benchmarking classification algorithms (1999).

Dataset of 60,000 28x28 grayscale images of the 10 digits,

Test set of 10,000 images.

CIFAR-10

Total 60,000 (50k + 10k) tiny images with dimension 32x32 pixels

Labeled with one of 10 classes (for example “airplane, automobile, bird, etc”).

​

​

Over 15M labeled high resolution images

• Roughly 22K categories

• Collected from web and labeled by Amazon Mechanical Turk

ILSVRC Task 2: Classification + Localization

ILSVRC Task 3: Detection

Long tailed Dataset

https://medium.com/@macaodha/inaturalist-2018-species-classification-challenge-254601e60600

​

Models

Ventral visual pathway

• Mediates Object recognition in cortex
• V1 - V2 - V4 - IT → PFC
• Cells found in macaque IT - key role in object recognition
• Hallmark of those cells - Robustness of their responses to stimulus transformations such as scale and position changes

Key Goals of Object Recognition

● Invariance - the ability to recognize a pattern under various transformations ● Specificity - the ability to discriminate between different patterns.

Neocognitron Fukushima (1980). Hierarchical multilayered neural network

• Multilayer neural network inspired by the mammalian visual system for handwritten digit classification
• Unsupervised image classification, tolerant to shifts and deformations
• Input - unlabeled images
• Output - vector, with each bit encoding a distinct class of images
• Generalizing simple-complex cells: alternating S and C layers: S - feature detectors (e.g. simple cells) detect conjunction of features (AND, specificity), C - invariance pooling (e.g. complex cells) (OR, MAX, invariance).

​

Neocognitron Fukushima (1980). Hierarchical multilayered neural network

​

Character Recognition

Hierarchical model and X (HMAX)

http://maxlab.neuro.georgetown.edu/hmax.html#c2

​

Hierarchical model and X (HMAX)

• A hierarchical build-up of invariances first to position and scale and then to viewpoint and more complex transformations requiring the interpolation between several different object views.
• In parallel, an increasing size of the receptive fields.
• An increasing complexity of the optimal stimuli for the neurons.
• A basic feedforward processing of information (for "immediate" recognition tasks)

​

AlexNet (2012 Winner, 15.3% error rate)

• Used ImageNet data (1.5M images, 1k classes) -- winner 2012
• Prior to this error rate was 25% and they brought it down to 15.3
• 8 layers where 5 are convolutional layers and 3 fully-connected layers
• ReLU activation function to add nonlinearity and improve the convergence rate
• leveraged multiple GPUs for faster training
• 60 million parameters making it susceptible to overfitting → dropout

AlexNet

ZFNet (2013 Winner, 11.2% error rate)

ZFNet used 7×7 sized filters, in contrast to 11x11 used in AlexNet to avoid the loss of pixel information.

​

GoogLeNet (2014 winner, 6.67% error rate)

• 22 layers

• Introduces a new layer architecture

​

GoogLeNet

VGG

• 19 layers
• Fixed filter size: 3x3
• Convolution Layer filters size 3 X 3 and stride = 1 and
• Max-pooling Layer has filters of size 2 X 2 and stride = 2.
• 140 millions parameters

VGG

Results on ILSVRC-2014

ResNet 2015

Practical Session

Colab Tutorial

Train MNIST:

https://colab.research.google.com/drive/1R2nEz93vMQ9fHjZDsBOIjfnVRPL0P7Sy

Colab Tutorials (Advanced)

https://colab.research.google.com/drive/1pN_VRJLhuEr4qjPRQTTtNNIFYELHN52f?usp=sharing

https://colab.research.google.com/drive/1Ka2icb6k0EwQ9-rqfckXnOyGyR6wNqhO?usp=sharing

​

​

Appendix

Trends

CIFAR 100

100 classes containing 600 images each.

There are 500 training images and 100 testing images per class.

The 100 classes in the CIFAR-100 are grouped into 20 superclasses.

Each image comes with a "fine" label (the class to which it belongs) and a "coarse" label (the superclass to which it belongs).

CIFAR 100

The PASCAL Visual Object Classes

The PASCAL VOC project:

• Provides standardised image data sets for object class recognition
• Provides a common set of tools for accessing the data sets and annotations
• Enables evaluation and comparison of different methods
• Ran challenges evaluating performance on object class recognition (from 2005-2012, now finished)

​

PASCAL Visual Object Challenge

Caltech 101

​

​

• Pictures of objects belonging to 101 categories.
• About 40 to 800 images per category. Most categories have about 50 images.
• Collected in September 2003 by Fei-Fei Li, Marco Andreetto, and Marc 'Aurelio Ranzato.
• The size of each image is roughly 300 x 200 pixels.

http://www.vision.caltech.edu/Image_Datasets/Caltech101/#Description

​

Large Scale Visual Recognition Challenge

ILSVRC

Medical Image Classification Datasets

​

​

1. Recursion Cellular Image Classification – This data comes from the Recursion 2019 challenge. This goal of the competition was to use biological microscopy data to develop a model that identifies replicates.

2. TensorFlow patch_camelyon Medical Images – This medical image classification dataset comes from the TensorFlow website. It contains just over 327,000 color images, each 96 x 96 pixels. The images are histopathological lymph node scans which contain metastatic tissue.

​

Agriculture and Scene Datasets

​

​

3. CoastSat Image Classification Dataset – Used for an open-source shoreline mapping tool, this dataset includes aerial images taken from satellites. The dataset also includes meta data pertaining to the labels.

4. Images for Weather Recognition – Used for multi-class weather recognition, this dataset is a collection of 1125 images divided into four categories. The image categories are sunrise, shine, rain, and cloudy.

5. Indoor Scenes Images – From MIT, this dataset contains over 15,000 images of indoor locations. The dataset was originally built to tackle the problem of indoor scene recognition. All images are in JPEG format and have been divided into 67 categories. The number of images per category vary. However, there are at least 100 images for each category.

Agriculture and Scene Datasets

​

​

6. Intel Image Classification – Created by Intel for an image classification contest, this expansive image dataset contains approximately 25,000 images. Furthermore, the images are divided into the following categories: buildings, forest, glacier, mountain, sea, and street. The dataset has been divided into folders for training, testing, and prediction. The training folder includes around 14,000 images and the testing folder has around 3,000 images. Finally, the prediction folder includes around 7,000 images.

7. TensorFlow Sun397 Image Classification Dataset – Another dataset from Tensorflow, this dataset contains over 108,000 images used in the Scene Understanding (SUN) benchmark. Furthermore, the images have been divided into 397 categories. The exact amount of images in each category varies. However, there are at least 100 images in each of the various scene and object categories.

Other Image Classification Datasets

​

​

8. Architectural Heritage Elements – This dataset was created to train models that could classify architectural images, based on cultural heritage. It contains over 10,000 images divided into 10 categories. The categories are: altar, apse, bell tower, column, dome (inner), dome (outer), flying buttress, gargoyle, stained glass, and vault.

9. Image Classification: People and Food – This dataset comes in CSV format and consists of images of people eating food. Human annotators classified the images by gender and age. The CSV file includes 587 rows of data with URLs linking to each image.

10. Images of Cracks in Concrete for Classification – From Mendeley, this dataset includes 40,000 images of concrete. Each image is 227 x 227 pixels, with half of the images including concrete with cracks and half without.

​

Other database

http://homepages.inf.ed.ac.uk/rbf/CVonline/Imagedbase.htm

​

Neocognitron Fukushima (1980). Hierarchical multilayered neural network

S-cells work as feature-extracting cells. They resemble simple cells of the primary visual cortex in their response.

C-cells, which resembles complex cells in the visual cortex, are inserted in the network to allow for positional errors in the features of the stimulus. The input connections of C-cells, which come from S-cells of the preceding layer, are fixed and invariable. Each C-cell receives excitatory input connections from a group of S-cells that extract the same feature, but from slightly different positions. The C-cell responds if at least one of these S-cells yield an output.

​

​