2 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Module-3 : Convolutional Networks

A Convolutional Neural Network (CNN) is a type of neural network mainly used to process grid-like data, especially images.
CNNs were introduced by Yann LeCun (1989).
They are widely used for image classification, face recognition, object detection, etc.
CNNs use an operation called convolution- mathematical operation which applies a small filter/kernel over the input to extract useful features.
Convolution is a special type of linear operation.
CNNs use convolution instead of general matrix multiplication in at least one layer.

3 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

Data can be represented as a grid:
1-D grid: Time-series data → values arranged according to time.
2-D grid: Image → pixels arranged in rows and columns.
3-D data: Colour image → height × width × channels (RGB).

A normal neural network connects every input pixel to every neuron. This creates a large number of parameters.
CNN instead looks at small regions of the image using a filter/kernel.
The filter can learn features such as: Edges, Corners, Textures, Shapes, Objects

4 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

The Convolution Operation

Convolution is a mathematical operation performed on two functions.

x(t) → Input function
w(t) → Weighting function (kernel/filter)
s(t) → Output of convolution

Example : Suppose we are tracking the location of a spaceship with a laser sensor.

We use a laser sensor to continuously measure its position, laser sensor provides a single output x(t)

x(t) = position of the spaceship at time t

Here: t = time, x(t) = position at that particular time

5 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

Suppose that our laser sensor is somewhat noisy, it produces some incorrect readings.

To obtain a less noisy estimate of the spaceship’s position, we would like to average several measurements.

More recent measurements are more relevant

To give different importance to measurements, we introduce a function:

w(a) Here: w = weighting function a = age of the measurement

So, recent measurements receive higher weights, while older measurements receive lower weights.

Suppose we want to calculate the spaceship's position at the current time t.

The measurement was taken at time a.
Then: t−a
tells us how old that measurement is.

6 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

If we apply such a weighted average operation at every moment, we obtain a new function providing a smoothed estimate of s the position of the spaceship:

s(t) - This is the smoothed estimate of the spaceship's position at time t.

x(a) - This represents the measurement of the spaceship's position at time a.

w(t−a) - This represents the weight given to that measurement based on its age.

da - This means that we consider measurements over all possible times and combine them.

This operation is called convolution. The convolution operation is typically denoted with an asterisk.

7 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

8 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

The 2-D convolution formula is

Convolution is commutative

The order of convolution does not change the mathematical result.

Many neural network libraries implement a related function

called the cross-correlation, which is the same as convolution

but without flipping the kernel

Apply the kernel to different locations of the image and

calculate an output value at each location.

9 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Motivation

Why convolution is useful in CNNs. The main idea is that convolution makes neural networks more efficient than a traditional fully connected layer

There are three important ideas

Sparse interactions
Parameter sharing
Equivariant representations

Sparse interactions

Traditional neural networks use matrix multiplication

Every input unit is connected to every output unit.
Therefore, a separate parameter/weight is required for each input–output connection.
If there are m inputs and n outputs, the number of parameters is: m x n

10 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

Convolutional neural networks (CNNs) use sparse interactions or sparse connectivity.

Sparse connectivity is achieved by using a kernel smaller than the input

The kernel interacts with only a small local region of the input at a time

For images, small kernels can detect meaningful features such as: Edges, Corners, Textures, Simple patterns

Advantages of Sparse Connectivity

Fewer parameters need to be stored.
Less memory is required.
Less computation is required.
The model becomes more statistically efficient.
The output can be computed using fewer operations.

11 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

Dense connectivity – When s is formed by matrix multiplication,

connectivity is no longer sparse, so all the outputs are affected by x3.

Sparse connectivity - Tne input unit, x3, and highlight the output units

in s that are affected by this unit.

When s is formed by convolution with a kernel of width 3,

only three outputs are affected by x.

12 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

When s is formed by matrix multiplication, connectivity is no

longer sparse, so all the inputs affect s3.

Sparse connectivity - one output unit, s3, and

highlight the input units in x that affect this unit.

These units are known as the receptive field of s3.

When s is formed by convolution with a kernel of width 3,

only three inputs affect s3

13 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Parameter sharing

Parameter sharing means using the same parameter (weight) in more than one place in a neural network.
In a traditional/dense neural network - Each input is connected to each output.
Each connection has its own weight.
So, weights are generally used only once for a particular input-output connection

In a CNN, the same kernel/filter weights are reused at different positions of the input
This is also called tied weights because the weights at different locations are tied to the same values.
Therefore, CNN does not learn a separate kernel for every location.
It learns one set of kernel parameters.
That set is reused everywhere.
Dense layer may need approximately m × n parameters, while a convolution needs only k parameters for the kernel.
Usually, k is much smaller than m × n, so CNN requires much less memory.

14 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued..

Below figures are showing parameter sharing in convolution and comparing it with a fully connected neural network.

The black arrows indicate uses of the central

element of a 3-element kernel in a convolutional model.

Because of parameter sharing, this single parameter is used at all input locations.

The single black arrow indicates the use of the central element

of the weight matrix in a fully connected model.

This model has no parameter sharing, so the parameter is used only once.

15 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Equivariant representations

A representation is the way a neural network describes or represents the input using its internal features.
For example, in an image, a convolutional layer may represent edges, corners, or shapes.
A representation is equivariant if a change in the input causes a corresponding change in the representation.
If f is the representation function and g represents a transformation such as translation:

This means:

First transform the input and then find its representation
OR first find the representation and then transform it

This is useful because objects can appear at different locations in an image.
We want the network to detect the same feature regardless of where it appears.
Convolution achieves this through parameter sharing.

16 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Pooling

Pooling is a technique used in CNNs to summarize information from a small neighborhood of pixels/features.
It takes several nearby values and replaces them with one representative value

A convolutional layer generally has three stages

Convolution 2. Activation / Detector 3. Pooling

In the first stage he CNN applies several filters at the same time. Each filter produces an output called a linear activation.

Linear = multiply the inputs by weights and add them together

In the second stage, each linear activation is run through a nonlinear activation function, such as the rectified linear activation function

After convolution and activation, we perform pooling to summarize the detected features.

17 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

Pooling makes the representation approximately translation invariant
So small movements don't change the pooled output very much.
Pooling helps the CNN become less sensitive to this small movement.

Types of Pooling

Max Pooling

Selects the largest value : y = Max(x₁,x₂,…,x_n)

Average Pooling

Takes the average.

L2 Pooling

Uses the L2 norm:

Weighted Average Pooling

Nearby values can be given different importance.
Values closer to the center can receive higher weights.

18 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

Pooling also reduces the size, Next CNN layer has fewer values to process.This is called downsampling.

19 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

Pooling helps to make the representation approximately invariant to small translations of the input

Invariance to translation means that if we translate the input by a small amount, the values of most of the pooled outputs do not change.

Max pooling introduces invariance it makes a CNN less sensitive to small shifts in an object.

20 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

Below figure is explaining how max pooling can make a CNN invariant to small changes in an object, here a handwritten digit 5

21 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

Pooling with downsampling

Here we use max pooling with a pool width of three and a stride between pools of two.
This reduces the representation size by a factor of two, which reduces the computational and statistical burden on the next layer.

22 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Examples of architectures for classification with convolutional networks.

23 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Variants of the Basic Convolution Function

Convolutional layers in deep learning have different variants depending on how the convolution operation is applied

1. Full Convolution: Expands the input by padding it with zeros, ensuring the output is larger than the input. This helps in capturing more context around the edges.

2. Valid Convolution: No padding is added, so the output is smaller than the input because only fully overlapping parts are computed.

3. Same Convolution: Pads the input to maintain the same output size as the input.

4. Unshared Convolution: Uses different filters for each location in the input, rather than sharing the same filter across all positions. This increase flexibility but requires more parameters.

5. Tiled Convolution: Reuse filters at intervals instead of applying them at every position, balancing parameter sharing and diversity.

These variations provide trade-offs between computational cost, memory, and the ability to capture features.

24 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Full Convolution

Full convolution means the kernel is allowed to overlap the input at every possible position, including positions where the kernel is partly outside the input.

Full convolution adds enough zeros around the input so that the kernel can slide beyond the original boundary

Example : Input: Kernel:

For full convolution, we allow the 2x2 kernel to go beyond the input boundary.
So conceptually we add one layer of zeros around the input, Then the 2x2 kernel slides across it.

The output becomes 4x4

Full output = Input size +Kernel size -1

25 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Convolution operations

Operation / Type	Kernel Flip?	Padding	Padding Formula	Output Size Formula
Valid Cross-Correlation	❌ No	❌ No
Same Cross-Correlation	❌ No	✅ Yes
Full Cross-Correlation	❌ No	✅ Yes
Valid Convolution	✅ Yes	❌ No
Same Convolution	✅ Yes	✅ Yes
Full Convolution	✅ Yes	✅ Yes
Unshared Convolution	✅/❌ Depends	Depends	Depends	Depends
Tiled Convolution	✅/❌ Depends	Depends	Depends	Depends

26 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

Example :

27 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Data Types

What different kinds of data can a convolutional network process, and how are those data represented?

CNN can work with different kinds of data

Data type	Example	Data structure
1-D single channel	Audio waveform	Time
1-D multichannel	Skeleton animation	Time + joint angles
2-D single channel	Fourier audio	Frequency × Time
2-D multichannel	Color image	Height × Width × RGB
3-D single channel	CT scan	Height × Width × Depth
3-D multichannel	Color video	Time × Height × Width × RGB

28 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

Earlier, we considered CNN inputs where all training and testing examples have the same size.

Example:

Image 1 → 224 × 224 Image 2 → 224 × 224 Image 3 → 224 × 224

CNNs have an advantage: they can also process different spatial sizes.

Example:

Image 1 → 224 × 224 Image 2 → 300 × 400 Image 3 → 512 × 512

Traditional fully connected neural networks use a fixed-size weight matrix, so they generally require a fixed input size.
CNNs use the kernel/filter that slide it across the input.
Therefore, the same CNN can process inputs with different spatial dimensions.
This is an important advantage of CNNs, even when computational cost and overfitting are not major concerns.

29 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

For example, consider a collection of images in which each image has a different width and height

Example: Image 1 → 100 × 100 Image 2 → 200 × 300 Image 3 → 500 × 400

Problem with traditional neural networks

A fully connected neural network uses a fixed-size weight matrix.
Therefore, it is difficult to directly handle images of different sizes

CNN handles this easily

A CNN uses a kernel/filter.
The same kernel slides over the image.
The number of times it slides depends on the image size.

30 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Continued…

Sometimes the output of the network as well as the input is allowed to have variable size.
for example, if we want to assign a class label to each pixel of the input. In this case, no further design work is necessary.

In other cases, the network must produce some fixed-size output.
for example, if we want to assign a single class label to the entire image.
In this case, we must make some additional design steps, like inserting a pooling layer whose pooling regions scale in size proportional to the size of the input, to maintain a fixed number of pooled outputs.

31 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Convolutional Networks and the History of Deep Learning

Convolutional Neural Networks (CNNs) played a major role in the development and success of deep learning.

CNNs were inspired by how the human brain processes visual information.

CNNs were among the first deep neural networks that worked successfully when deep networks were still difficult to train.

CNNs were used in important real-world commercial applications even before the recent deep-learning boom

In the 1990s, researchers at AT&T developed CNNs for reading handwritten checks.

By the end of the 1990s, the system deployed by NCR was reportedly reading more than 10% of checks in the United States.

CNN-based systems were later used by Microsoft for:

- Optical Character Recognition (OCR)

- Handwriting recognition

32 of 32

27/07/2026

/skit.org.in

(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)

Sri Krishna Institute of Technology

Convolutional Networks and the History of Deep Learning

CNNs won several machine learning and computer vision competitions even before 2012.
In 2012, Alex Krizhevsky and colleagues used a CNN to win the ImageNet object recognition challenge.
This significantly increased commercial interest in deep learning.
CNNs were among the first practical deep networks successfully trained using backpropagation.
CNNs are more computationally efficient than fully connected networks
Larger neural networks may actually be easier to train with modern hardware and techniques.
CNNs demonstrated that neural networks could solve practical problems.
CNNs are particularly suitable for data with a clear grid structure.

Example: an image is a 2D grid of pixels.

For one-dimensional sequential data, such as time-series or sequences, the text moves to another neural-network specialization: Recurrent Neural Networks (RNNs).