Artificial Neural Networks (ANN) and Convolutional Neural Networks (CNN): Image Processing

Housing Price Prediction

size of house

price

Housing Price Prediction

Housing Price Prediction

#bedrooms

zip code

wealth

size

y

Supervised Learning

Output (y)

​

Application

​

Input(x)

​

Neural Network examples

Structured Data

Supervised Learning

Unstructured Data

Neural Networks

• Subset of machine learning models that are inspired from the working of biological neurons
• The “building blocks” of neural networks are the neurons.
• In technical systems, we also refer to them as units or nodes.
• Basically, each neuron
• receives input from many other neurons.
• changes its internal state (activation) based on the current input.
• sends one output signal to many other neurons, possibly including its input neurons (recurrent network).

How do our brains work?

• A processing element
• Dendrites: Input
• Cell body: Processor
• Synaptic: Link
• Axon: Output
• A neuron is connected to other neurons through about 10,000 synapses
• A neuron receives input from other neurons. Inputs are combined
• Once input exceeds a critical level, the neuron discharges a spike ‐ an electrical pulse that travels from the body, down the axon, to the next neuron(s)

Artificial Neurons

• Inspired from Biological Neurons

​

• Information is received from multiple inputs, processed and a combined output is generated.

​

• Does not mimic all the functionality of biological neuron, it is a highly restrictive model

Dendrites: Input

Cell body: Processor

Axon terminals (Synaptic): Link

Axon: Output

An artificial neuron is an imitation of a human neuron

Artificial Neurons

• Now, let us have a look at the model of an artificial neuron.

How do ANNs work?

Output

x1

x2

xm

∑

y

Processing

Input

∑= X1+X2 + ….+Xm =y

. . . . . . . . . . . .

How do ANNs work?

Not all inputs are equal

Output

x1

x2

xm

∑

y

Processing

Input

∑= X1w1+X2w2 + ….+Xmwm =y

w₁

w₂

w_m

weights

. . . . . . . . . . . .

. . . . .

How do ANNs work?

The signal is not passed down to the next neuron verbatim

Transfer Function (Activation Function)

Output

x1

x2

xm

∑

y

Processing

Input

w₁

w₂

​

w_m

weights

. . . . . . . . . . . .

f(v_k)

. . . . .

How do ANNs work?

• Inputs are multiplied with weights

​

• Weighted inputs are added together with bias

​

• The sum is passed through activation function f

Simple Neural Network

Neural Network with 2 inputs, 2 hidden nodes and one output

Activation Functions

• Activation functions keep the output of a neuron restricted to a certain limit

​

• Important function is to introduce non-linearity into the neural network

Sigmoid

ReLU (Rectified Linear Unit)

Activation Functions

Sigmoid

ReLU (Rectified Linear Unit)

Threshold

Tanh

Example

Goal is to find the function y

in terms of x1 and x2

• We are given the dataset on the right

​

• Let’s solve this using neural networks!

​

Simple Neural Network

Neural Network with 2 inputs, 1 hidden nodes and one output

Training a Neural Network

• Loss function
• Loss function helps us understand ”How good the model is doing”. One way to evaluate our model

​

• Mean Squared Error (Regression Problems)

​

• Cross Entropy Loss / Negative Log Likelihood (Classification problems)

​

Backpropagation

How to modify weights given the loss, so that loss can be minimized

• Loss function can be written in terms of the weights and biases

​

• Calculate how much loss changes when we tweak the weights (learnable parameters)

​

• The system of calculating partial derivatives by working backwards is called backpropagation

Optimization – Gradient Descent

How to update the weights?

• Optimization function tells us how to update the weights and biases to minimize the loss

​

• n – learning rate that controls how fast we train. It is a hyper parameter.

​

• A model hyperparameter is a configuration that is external to the model and whose value cannot be estimated from data.

​

Gradient Descent intuition

Loss function graphed

We want to find the value of

weights such that loss is minimized

Neural Network Training – Put it all together

• 1. Choose a sample data point from the dataset.

​

• 2. Pass the input through the network

​

• 3. Calculate all the partial derivatives of loss with respect to weights or biases (e.g. ∂L/∂w1​, ∂L​/∂w2​, etc).

​

• 4. Use the optimization equation to update each weight and bias.

​

• 5. Go back to step 1.

Step 1: Feed Forward Step

For simplicity, assume initial weights are 0.5 and biases are 0.

Let’s take activation function as ReLU

Start with first sample from dataset

Step 2: Calculate Loss and Backpropagation

Let’s take Loss function as MSE

Step 3: Update weights with gradient descent

Step 4: Repeat the process

• Now we repeat the process with other samples from the dataset and update the weights

​

• We can stop this if our loss value is zero, then we approximated the function y

​

• Or, we can stop after a set number of maximum iterations. (eg. 100 max iterations)

Deep neural networks

• Neural network with more than 2 layers
• DNNs are a specific type of ANN characterized by their depth, meaning they have many hidden layers.
• Can model more complex functions

Anatomy of a deep neural networks

​

• Layers
• Input data and targets
• Loss function
• Optimizer

Layers

• Data processing modules
• Many different kinds exist
• densely connected
• convolutional
• recurrent
• pooling, flattening, merging, normalization, etc.
• Input: one or more tensors�output: one or more tensors
• Usually have a state, encoded as weights
• learned, initially random
• When combined, form a network or�a model

Input data and targets

​

• The network maps the input data X to predictions Y′
• During training, the predictions Y′ are compared to true targets Y using the loss function

cat

dog

Loss function

• The quantity to be minimized (optimized) during training
• the only thing the network cares about
• there might also be other metrics you care about
• Common tasks have “standard” loss functions:
• mean squared error for regression
• binary cross-entropy for two-class classification
• categorical cross-entropy for multi-class classification
• etc.
• https://lossfunctions.tumblr.com/

Optimizer

• How to update the weights based on the loss function
• Learning rate (+scheduling)
• Stochastic gradient descent, momentum, and their variants
• RMSProp is usually a good first choice
• more info: http://ruder.io/optimizing-gradient-descent/

Animation from: https://imgur.com/s25RsOr

Anatomy of a deep neural networks

Smaller Network: CNN

• We know it is good to learn a small model.
• From this fully connected model, do we really need all the edges?
• Can some of these be shared?

A CNN arranges its neurons in three dimensions (width, height, depth). Every layer of a CNN transforms the 3D input volume to a 3D output volume. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels).

A regular 3-layer Neural Network.

Consider learning an image:

• Some patterns are much smaller than the whole image

“beak” detector

Can represent a small region with fewer parameters

Same pattern appears in different places:�They can be compressed!�What about training a lot of such “small” detectors�and each detector must “move around”.

“upper-left beak” detector

“middle beak” detector

They can be compressed

to the same parameters.

What are Convolutional Neural Networks (CNNs)

• CNNs are similar to MLPs since they only feed signals forward (feedforward nets), but have different kind of layers unique to CNNs
• Convolutional Layer: process data in a small receptive field (i.e., filter)
• Pooling layer: Pooling layers perform spatial down-sampling by selecting the most important information from the feature maps produced by the convolutional layers
• Dense (fully connected) layer: These layers consolidate high-level features and make final decisions

A simple CNN structure

CONV: Convolutional kernel layer

RELU: Activation function

POOL: Dimension reduction layer

FC: Fully connection layer

Convolutional Layer

• Convolutional layers apply a set of learnable filters to input data to extract meaningful features
• Filters also known as kernels
• Designed to detect patterns and hierarchies of features
• Learnable Filters (Kernels)
• these filters are adjusted through backpropagation to capture relevant features in the data
• learn the most informative patterns

Convolutional Kernels

• A convolutional kernel, often referred to as a filter or convolutional filter, is a small matrix used for feature extraction
• Kernels are applied to the input data through convolution operations
• Kernels serve as feature detectors
• Each kernel specializes in detecting a specific pattern or feature within the input data
• For example, one kernel might detect edges, while another could detect corners
• Learnable Parameters
• Kernels contain learnable parameters that are adjusted during training
• These parameters determine the filter's behavior and help it adapt to detect relevant features in the data
• Size
• Kernels are typically square and have a small size, such as 3x3 or 5x5
• The size of the kernel affects the receptive field

Kernel

Feature Maps

• Feature map
• also known as an activation map or a convolutional feature map
• A two-dimensional grid of values resulting from the application of a convolutional kernel to the input data
• Each feature map corresponds to a specific kernel
• It represents the presence or activation of particular features
• A way to visualize which patterns or structures the network has detected
• The number of feature maps in a layer is determined by the number of kernels or filters used

​

Feature Activation Map

What is a Convolution?

• Weighted moving sum

Input

Feature Activation Map

...

Convolution

6 x 6 image

Filter 1

Filter 2

……

These are the network parameters to be learned.

Each filter detects a small pattern (3 x 3).

Convolution

6 x 6 image

Filter 1

3

-1

stride=1

Dot

product

Convolution

6 x 6 image

Filter 1

3

-3

If stride=2

Convolution

6 x 6 image

Filter 1

3

-1

-3

-1

-3

1

0

-3

-3

-3

0

1

3

-2

-2

-1

stride=1

Convolution

6 x 6 image

3

-1

-3

-1

-3

1

0

-3

-3

-3

0

1

3

-2

-2

-1

Filter 2

-1

-1

-1

-1

-1

-1

-2

1

-1

-1

-2

1

-1

0

-4

3

Repeat this for each filter

stride=1

Two 4 x 4 images

Forming 2 x 4 x 4 matrix

Feature

Map

image

convolution

……

……

Convolution v.s. Fully Connected

Fully-connected

6 x 6 image

Filter 1

1

2

3

…

8

9

…

13

14

15

…

Only connect to 9 inputs, not fully connected

4:

10:

16

1

0

0

0

0

1

0

0

0

0

1

1

3

fewer parameters!

Filter 1

1:

2:

3:

…

7:

8:

9:

…

13:

14:

15:

…

4:

10:

16:

1

0

0

0

0

1

0

0

0

0

1

1

3

-1

Shared weights

6 x 6 image

Fewer parameters

Even fewer parameters

A CNN compresses a fully connected network in two ways:

• Reducing number of connections
• Shared weights on the edges
• Max pooling further reduces the complexity

Color image: RGB 3 channels

Filter 1

Filter 2

ReLU Layer

• ReLU is an activation function
• Activation functions are typically applied after each convolutional layer in CNNs
• The main aim is to remove all the negative values from the convolution
• Activation functions introduce non-linearity,
• allowing the network to model complex data patterns and relationships effectively

3

0

0

0

0

1

0

0

0

0

0

1

3

0

0

0

Pooling layer

• Pooling layers
• also known as pooling operations
• crucial component in CNNs
• used for feature extraction and dimensionality reduction
• These layers are inserted between convolutional layers
• downsample feature maps
• reducing their spatial dimensions
• retain important information
• Pooling is performed by implementing the following 4 steps
• Pick a window size (usually 2 or 3)
• Pick a stride (usually 2)
• Walk your window across your filtered images
• From each window, take the maximum value
• Pooling helps reduce computational complexity, mitigate overfitting, and increase the depth of CNNs
• In general, max pooling method is used

Max Pooling

3

-1

-3

-1

-3

1

0

-3

-3

-3

0

1

3

-2

-2

-1

Filter 2

-1

-1

-1

-1

-1

-1

-2

1

-1

-1

-2

1

-1

0

-4

3

Filter 1

Why Pooling

• Subsampling pixels will not change the object

Subsampling

bird

bird

We can subsample the pixels to make image smaller

fewer parameters to characterize the image

Key characteristics and functions of pooling layers

• Downsampling
• Pooling layers reduce the spatial dimensions (width and height) of the feature maps
• Translation Invariance
• Pooling layers help achieve translation invariance in the learned features
• Dimension Reduction
• By reducing the spatial dimensions of feature maps, pooling layers reduce the number of parameters in the subsequent layers

The whole CNN

cat dog ……

Convolution

Max Pooling

Convolution

Max Pooling

Flattened

Can repeat many times

Max Pooling

6 x 6 image

3

0

1

3

-1

1

3

0

2 x 2 image

New image

but smaller

Conv

Max

Pooling

The whole CNN

Convolution

Max Pooling

Convolution

Max Pooling

Can repeat many times

A new image

The number of channels is the number of filters

Smaller than the original image

The whole CNN

cat dog ……

Convolution

Max Pooling

Convolution

Max Pooling

Flattened

A new image

A new image

Flattening

Flattened

3

0

1

3

-1

1

0

3

Fully Connected Layers

• CNNs primarily consist of convolutional and pooling layers for feature extraction
• fully connected layers are typically found at the end of the network
• Feature Aggregation
• Aggregate and process the features extracted by the preceding convolutional and pooling layers.
• In the context of tasks like image classification, fully connected layers make the final decisions
• They map the learned features to the output classes or values, producing the network's predictions
• The final fully connected layer typically has as many neurons as there are output classes
• The softmax activation function is often used in this layer to produce class probabilities

Hierarchical Feature Extraction

Retain most information (edge detectors)

Towards more abstract representation

Encode high level concepts

Sparser representations:

Detect less (more abstract) features

Data Preparation

Basic CNN model definition

Model summary

Training

Evaluation