MACHINE LEARNING

Dr. P V Siva Teja

Assistant Professor

Dr. P V Siva Teja

UNIT-IV: Linear Discriminants for Machine Learning�(Normal Discriminant Analysis (or) Discriminant Function Analysis)

Introduction to Linear Discriminants, Linear Discriminants for Classification, Perceptron Classifier, Perceptron Learning Algorithm, Support Vector Machines, Linearly Non-Separable Case, Non-linear SVM, Kernel Trick, Logistic Regression, Linear Regression, Multi-Layer Perceptron’s (MLPs), Backpropagation for Training an MLP.

Dr. P V Siva Teja

Dr. P V Siva Teja

Discriminant

The name "discriminant" comes from its ability to "discriminate" or distinguish b/w the different types of solutions an equation can have, specially whether they are:

• Real and distinct
• Real and equal or complex.

In math

The discriminant is a value calculated from the coefficients of a polynomial equation that reveals the nature of its roots (solutions) without needing to solve the entire equation.

For a quadratic equation:

ax² + bx + c = 0

Discriminant = b² − 4ac (or) using standard notation: D = b² − 4ac

Dr. P V Siva Teja

The value of the discriminant (D = b² − 4ac) determines the characteristics of the roots of a quadratic equation without the need to solve the entire equation. There are three distinct types of solutions based on this value:

• Real and Distinct: This occurs when D > 0. In this case, the equation has two different real-number solutions.
• Real and Equal: This occurs when D = 0. In this case, the equation has two identical real-number solutions (also known as a repeated root).
• Complex: This occurs when D < 0. In this case, the equation has no real solutions; instead, the solutions are imaginary or complex numbers.

Dr. P V Siva Teja

Example: Finding the Nature of Roots

Equation:�x² − 5x + 6 = 0

Step 1: Identify the coefficients In the standard form ax² + bx + c = 0:�a = 1 b = −5 c = 6

Step 2: Apply the Discriminant formula�The formula for the discriminant (D) is: D = b² − 4ac

Step 3: Calculate the value

D = (−5)² − 4(1)(6) = 25 − 24 = 1

Step 4: Determine the Nature of Roots�Since D = 1 (which is greater than 0), the value of the discriminant is positive (D > 0).�Therefore, the equation has real and distinct roots.

Introduction to Linear Discriminant Analysis (LDA)

• It is a supervised technique used for both dimensionality reduction and classification.
• It works by finding a linear combination of features that maximizes the separation between multiple classes while minimizing the variation within those classes.
• LDA projects high-dimensional data (no. of attributes) onto a lower-dimensional space, making it more efficient for classification and aiding in data visualization.

Linear Discriminant Analysis Works

1. Supervised Approach

Unlike unsupervised methods like PCA (Principal Component Analysis), LDA is a supervised technique, meaning it uses class labels during the training process to guide its computations.

2. Maximizes Class Separability

LDA aims to find axes (linear discriminants) that maximize the ratio of between-class variance (difference between class means) to within-class variance (spread within each class).

3. Data Projection

The data is then projected onto these newly found axes, transforming the high-dimensional data into a lower-dimensional space where classes are as distinct as possible.

​

Assumptions

LDA assumes that the data follows a multivariate Gaussian distribution for each class and that all classes share the same covariance matrix.

Dr. P V Siva Teja

Linear Discriminant for Classification

• LDA is a dimensionality reduction and classification technique commonly used in ML and pattern recognition.
• It aims to find a linear combination of features that best separates different classes or categories of data. It seeks to reduce the dimensionality of the feature space while preserving as much of the class-separability information as possible.

Dr. P V Siva Teja

• Dimensionality reduction involves separating data points with a straight line. Mathematically, linear transformations are analyzed using eigenvectors and eigenvalues.
• Imagine you have mapped out a dataset with multiple features, resulting in a multi-dimensional scatterplot.
• Eigenvectors provide the "direction" within the scatterplot.
• Eigenvalues denote the importance of this directional data. A high eigenvalue means the associated eigenvector is more critical.

Dr. P V Siva Teja

LDA Algorithm Overview

LDA aims to find a projection (linear combination of features) that maximizes the separation between classes while minimizing the variance within each class. This makes classes more distinguishable in a lower-dimensional space.

Step-by-Step Algorithm

1. Data Preparation

• Dataset: 150 Iris samples.
• Features: 4 features per sample.
• Classes: 3 Species (Setosa, Versicolor, and Virginica), evenly distributed.

2. Compute Class Statistics

• Calculate the mean and covariance matrix for each feature in each class.
• Result: Three mean vectors and three covariance matrices (one for each class).

3. Compute Scatter Matrices

• Between-Class Scatter (Sb): Measures the differences between the mean vectors of each class and the overall mean, and summing these outer products.

Dr. P V Siva Teja

Dr. P V Siva Teja

Dr. P V Siva Teja

4. Compute Eigenvectors and Eigenvalues

• Solve the generalized eigenvalue problem using the between-class scatter matrix (Sb) and the within-class scatter matrix (Sw).
• This yields a set of eigenvectors and their corresponding eigenvalues.

5. Select Discriminant Directions

• Sort eigenvectors by their eigenvalues in descending order.
• Select the top k eigenvectors (e.g., reducing dimensionality to 2).

6. Transform Data

• Project the original Iris data onto the selected eigenvectors.
• This results in a new lower-dimensional representation (e.g., 2D).

7. Classification

• In the reduced-dimensional space, use a classifier (e.g., KNN) to classify samples based on their new positions.

Dr. P V Siva Teja

Dr. P V Siva Teja

• LDA aims to find the projection (linear combination of features) that maximizes the separation b/w the classes, while minimizing the variance within each class. This way, the classes become more distinguishable in the lower-dimensional space.

​

• In our Iris flower example, LDA would find the best linear combination of sepal length, sepal width, petal length and petal width that maximizes the separability b/w the Setosa, Versicolor and virginica species. The reduced-dimensional space could potentially help in better classifying new Iris samples.

Dr. P V Siva Teja

Advantages

• Dimensionality Reduction
• Improved Classification Performance
• Computational Efficiency
• Interpretability
• Works well with small datasets
• Handles Multi-class Classification
• Feature Selection
• Data Visualization

Dr. P V Siva Teja

Disadvantages

• Sensitivity to outliers
• (Gaussian) Overfit with high-dimensional data

Applications

Dr. P V Siva Teja

Perceptron

Perceptron is a simplest possible "brain cell" (neuron) for a computer. It is the fundamental building block of an Artificial Neural Network.

Its job is simple: It takes several inputs, processes them, and decides whether to "fire" (output a 1) or stay silent (output a 0).

​

​

​

Artificial Neural Network (ANN):-

• ANN is inspired by the working of a human brain.
• The human brain that has neurons interconnected to one another, ANN also have neurons, that are interconnected to one another in various layers of the networks. These neurons are known as nodes.
• Neural Networks are a set of algorithms that tries to recognize the patterns, relationships, and information from the data through the process which is inspired by and works like the human brain.

Dr. P V Siva Teja

Comparison of Neural Networks

Dr. P V Siva Teja

Components of ANN

A simple neural network consists of three components. They are:

1. Input layer
2. Hidden layer (middle layer)
3. Output layer

​

1. Input layer

Input nodes receive inputs / information from the outside world.

2. Hidden layer

Hidden layer is the set of neurons where all the computations are performed on the input data.

There can be any number of hidden layers in a neural network. The simplest network consists of a single hidden layer.

3. Output layer

The output layer is the output / conclusions derived from all the computations performed.

There can be single or multiple nodes in the output layer:

• Single node: Binary classification
• Multiple nodes: Multi-class classification

Dr. P V Siva Teja

Perceptron

It is a simple form of Neural Network and consists of a single layer where all the mathematical calculations/computations are performed.

Multilayer Perceptron (MLP)

It is also known as an ANN (Artificial Neural Network). It consists of more than one perceptron which is grouped together to form a multiple layer neural network.

Dr. P V Siva Teja

Perceptron Learning Algorithm

• One of the earliest and most fundamental algorithms in ML is the perceptron model.
• Developed by Frank Rosenblatt, the perceptron model is historically significant for laying the groundwork for neural networks and modern AI systems.
• It remains a foundational building block in the field of supervised learning and classification.

Dr. P V Siva Teja

Perceptron Model in ML

Perceptron Model is a type of artificial neuron that functions as a linear binary classifier. Its purpose is to classify data points into one of two categories by learning a decision boundary from labeled training data.

• The perceptron model calculates the weighted sum of input values, applies an activation function, and produces a binary output.
• If the output meets or exceeds a specific threshold, then it is classified into one category; else, it falls into the other category.

This process makes the perceptron particularly useful in tasks such as binary classification in datasets like spam detection or sentiment analysis.

Dr. P V Siva Teja

Basic Components of a Perceptron

1. Input Layer

The input layer consists of feature values or data points that the perceptron will classify. Each input is assigned a corresponding weight.

2. Weights

Weights determine the importance of each input in the classification process. The model adjusts these weights during training to improve accuracy.

3. Bias

The bias term helps the perceptron shift the decision boundary, improving flexibility in data classification.

4. Activation Function

The activation function, such as a step function or sigmoid function, determines the output based on the weighted sum of the inputs.

5. Output

The output is the final result of the perceptron's decision, typically a binary value (0 or 1) in the case of binary classification.

Dr. P V Siva Teja

Perceptron Algorithm

The perceptron model operates in a step-by-step process that involves computing the weighted sum of inputs, applying an activation function, and classifying the output. This process allows the model to differentiate between two classes based on the input data.

Step-by-Step Process

Step 1: Initialize Weights and Bias

The model begins by assuming random weights for the inputs and setting a bias value.

Step 2: Compute Weighted Sum

For each input, the perceptron computes the weighted sum of the input values and the bias term.

It is represented mathematically as:

Z = w₁x₁ + w₂x₂ + … + wₙxₙ + b

Where:�w = weights�x = inputs�b = bias

Dr. P V Siva Teja

Step 3: Apply Activation Function

The weighted sum is then passed through an activation function. If the result meets a certain threshold, the output is classified as 1. Otherwise, it is classified as 0. The step function is a common activation function used for this purpose:

​

f(Z) = 1, if Z ≥ 0�f(Z) = 0, if Z < 0

​

Step 4: Update Weights

During training, the model compares the predicted output to the actual output and adjusts the weights using a learning rule (typically gradient descent) to reduce the error.

Step 5: Repeat

This process is repeated for multiple iterations until the model's predictions align with the actual outputs.

​

Dr. P V Siva Teja

Advantages

• Simplicity: The model is easy to understand and implement, making it an excellent starting point for learning about neural networks.
• Efficiency: It is efficient when working with linearly separable data, as it quickly converges to a solution.

Limitations: The perceptron can only solve problems with linear decision boundaries, making it unsuitable for more complex tasks.

Applications

• Binary classification
• Image recognition
• Speech processing

Dr. P V Siva Teja

Support Vector Machines (SVM)

SVM is a Machine Learning (ML) algorithm used for classification and regression.

Goal or Aim

The goal of the SVM algorithm is to find the best line (or hyperplane) to separate data into groups, maximizing the distance between the closest points (Support Vectors) of each group.

Key Features

• It can handle complex data using Kernels to transform it into higher dimensions.
• SVM helps classify data effectively.

Dr. P V Siva Teja

SVM Diagram

Dr. P V Siva Teja

Types of SVM

SVM can be classified into two types:

1. Linear SVM
2. Non-linear SVM

​

1. Linear SVM:-

It is used for linearly separated data, which means if a dataset can be classified into two classes by using a single straight line, then such data is termed as linearly separable data, and classifier is used called as linear SVM classifier.

​

2. Non-linear SVM:-

It is used for non-linearly separated data, which means if a dataset cannot be classified by using a straight line, then such data is termed as non-linear data and classifier used is called as non-linear SVM classifier.

Dr. P V Siva Teja

Training datasets which have a significant grouping margin will function well with SVM.

The important terms used in SVM are:

• generalization error
• Support Vector
• hyperplane and margin
• Generalization error is the measure of how accurately an SVM model can predict new data.
• SVM model is inflexible in classification & tries to fit the hyperplane in the training set, thereby causing overfitting.
• Support vectors - Data points that are close to the hyperplane are called support vectors. The separating line will be defined with the help of these data points.

Dr. P V Siva Teja

Hyperplane

A hyperplane is a decision plane or space that is divided into a set of objects having different classes.

​

2-Dimensional Space: Mathematically, in a 2D space, a hyperplane can be defined by the equation of a straight line:

C0​+C1​x1​+C2​x2​=0

N-Dimensional Space: Extending this concept to an N-dimensional space, the hyperplane can be defined by the equation:

C0​+C1​x1​+C2​x2​+⋯+CN​xN​=0

Vector Representation: This can also be represented as:

C⋅X+C0​=0

Dr. P V Siva Teja

Margin

The margin is defined as the gap between two lines on the closest data points of different classes.

• Calculation: It is calculated as the perpendicular distance from the line to the support vectors.
• Large Margin: Considered a good margin.
• Small Margin: Considered a bad margin.

Dr. P V Siva Teja

Support Vector Classification Algorithm

The primary goal of this algorithm is to find the "best" line (or hyperplane) that separates two classes of data points with the largest possible gap.

Step 1: Select Initial Hyperplanes

• Action: Select two initial hyperplanes (represented by the dotted lines) that separate the data into two distinct groups.
• Condition: These lines should be placed such that there are no data points between them.

Dr. P V Siva Teja

Step 2: Maximize the Margin

• Action: Adjust the distance between these two dotted lines until it is as large as possible.
• Concept: This distance is known as the Margin. Maximizing this margin ensures that the classifier is "confident" and reduces the chance of misclassifying new, unseen data points.
• Technical Note: The data points that sit exactly on these dotted lines are called Support Vectors.

Dr. P V Siva Teja

Step 3: Define the Decision Boundary

• Action: Create an "average line" or a line that sits exactly halfway between the two dotted lines.
• Result: This central solid line is the Decision Boundary (or Optimal Hyperplane).
• Classification Rule: Anything on one side of this boundary belongs to Class A, and anything on the other side belongs to Class B.

Dr. P V Siva Teja

Advantages

• SVM can be used for both classification and regression.
• It is robust, i.e., not much impacted by data with noise or outliers.
• The prediction results using the model are very promising.

​

Disadvantages

• It is slow for a large dataset, i.e., a dataset with either a large number of features or a large no. of instances.
• It is quite memory-intensive.

​

Applications

• Face detection
• Text & hypertext categorization
• Classification of images
• Bioinformatics
• Handwriting recognition

Dr. P V Siva Teja

Linearly Non-Separable Case

Scenario 1 : Data is linearly separable.

• Let us consider 2 classes – apples & lemons.
• In contrast, SVM will search for apples that are very similar to lemons, for example: apples which are yellow and have an elliptic form.
• This will be a Support Vector.
• The other Support Vector will be a lemon similar to an apple (round & rounded).
• So, other algorithms learn the differences where SVM learns similarities.

Dr. P V Siva Teja

Finding the Optimal Hyperplane

• Based on these support vectors, the algorithm tries to find the best hyperplane that separates the classes.
• SVM will find those two boundary data points or support vectors.
• These support vectors should be at a maximum margin from the optimal hyperplane.
• SVM finds the two closest support vectors and separates the classes with the optimal hyperplane.

Dr. P V Siva Teja

Summary of the Optimal Solution

• Finding the optimal hyperplane: Intuitively, the best line is the line that is far away from both apple and lemon examples (has the largest margin).
• To have an optimal solution, we have to maximize the margin in both ways.
• All in all, Support Vectors are data points that define the position and the margin of the hyperplane.

Dr. P V Siva Teja

Scenario-2: Data is Non-linearly Separable (or) linearly Non-separable case

• In this case, we cannot find a st. line (straight line) to separate classes (Apples & Lemons).
• We will use the Kernel trick.
• The basic idea is that when a dataset is inseparable in the current dimensions, add another dimension; maybe that way the data will be separable.

Mapping — From 1D to 2D

• After using the kernel transformation is happen.
• So, after the transformation, we can easily delimit the two classes using just a single line.

​

Dr. P V Siva Teja

Non-linear SVM

• A non-linear SVM is a ML algorithm used when data cannot be separated by a st-line (straight line), employing kernel functions to map the data into a higher-dimensional space where it becomes linearly separable.
• It allows for complex patterns and curved decision boundaries to be found by the algorithm, improving classification accuracy for non-linearly separable datasets.

Dr. P V Siva Teja

Comparison: Linear SVM vs. Non-linear SVM

Dr. P V Siva Teja

Kernel trick in SVM

​

Kernel trick is a function, used to solve the non-linear Support Vector Machines problems.

• In the applications, the data can be text, image or video.
• So, there is a need to extract features from these data prior to classification.
• Hence, in the real world, many classification models are complex and mostly require non-linear hyperplanes.

Dr. P V Siva Teja

Mapping Function: 2D to 3D

�One mapping function φ: ℝ² → ℝ³ used to transform 2D data to 3D data is given as:

φ(x, y) = (x², √2xy, y²)

Example Case:�Consider a point (2, 3) in 2D space. If you apply the above mapping function, we can convert it into 3D space.:

φ(2, 3) = (2², √12, 3²) = (4, 3.46, 9)

​

Parameters:�x = 2�y = 3

Dr. P V Siva Teja

Mapping function plays an important role, there are many disadvantages:

• As mapping involves more computation
• High learning costs.

Kernels are used to overcome the above two disadvantages.

Kernels are used to compute the value without transforming the data.

Dr. P V Siva Teja

Kernel

• Kernels are a set of functions used to transform data from low dimension and to manipulate data using data product at highest dimensions.
• The use of kernel is to apply transformation to data and perform classification at a higher dimension as shown below:

Dr. P V Siva Teja

Types of Kernels:

​

Linear Kernel Polynomial Kernel

Gaussian RBF (Radial Basis Function) Sigmoid

Dr. P V Siva Teja

Dr. P V Siva Teja

Dr. P V Siva Teja

Logistic Regression

It is a supervised ML algorithm for binary classification, predicting the probability of an outcome (e.g., spam or not spam) by applying a sigmoid function to a linear combination of input features.

Dr. P V Siva Teja

Input : Output of the linear regression function.

Output : Uses a sigmoid function to estimate the probability for the given class.

It is used to describe data and to explain the relationship b/w one dependent binary variable and one or more nominal, ordinal, interval or ratio-level independent variables.

Dr. P V Siva Teja

Logistic Regression Works:

Step 1: Prepare the data

The data should be in a format where each row represents a single observation and each column represents a different variable. The target variable should be binary (yes/no, true/false).

Step 2: Train the Model

We teach the model by showing it the training data. This involves finding the value of the model parameters that minimize the error in the training data.

Step 3: Evaluate the Model

The model is evaluated on the held-out test data to assess its performance on unseen data.

Step 4: Use the model to make predictions

After the model has been trained and assessed, it can be used to forecast outcomes on new data.

Dr. P V Siva Teja

Types of Logistic Regression

On the basis of categories, Logistic Regression can be classified into three types:

1. Binomial Logistic Regression

This type is used when the dependent variable has only two possible outcomes.

• Examples: 0 or 1, Pass or Fail, etc.
• Prediction Tool: The Sigmoid function is used for prediction.

2. Multinomial Logistic Regression

This type is used when the dependent variable has 3 or more possible unordered outcomes.

• Examples: "Cat", "Dogs", or "Sheep".
• Prediction Tool: The Softmax function is used in place of the Sigmoid function for K classes.

3. Ordinal Logistic Regression

This type is used when the dependent variable has 3 or more possible ordered outcomes (where the sequence matters).

• Examples: "Low", "Medium", or "High".

​

​

Dr. P V Siva Teja

Advantages

• Linear Separability: It performs well when the dataset is linearly separable.
• Coefficient Insights: It provides a measure of how relevant a predictor is (coefficient size) and indicates the direction of association, whether positive or negative.
• Efficiency: It is easy to implement, simple to interpret, and very efficient to train.

​

Disadvantages

• Linearity Assumption: A main limitation is the assumption of linearity between the dependent and independent variables.
• Overfitting Risk: If the number of observations is less than the number of features, it should not be used because it may lead to overfitting.

​

Applications

Logistic Regression is widely used in various fields for classification tasks:

Spam Detection, Medical Diagnosis, Fraud Detection, Customer Churn Prediction, Loan Default Prediction, Product Purchase Prediction, Risk Assessment

​

Dr. P V Siva Teja

Assumptions for Logistic Regression

1. Linearity

A straight line (best fit line) should exist between the independent variables and the log odds of the dependent variable. Unlike linear regression, it does not require a linear relationship between the independent and dependent variables themselves.

2. Independence

The observations (outputs) should stand alone from one another.

• Implication: The response variable's value for a given observation should not be tied to its value for any other observation.
• Note: One input response (i/p) should be tied to one observation (o/p) only.

3. No Multicollinearity

There shouldn't be a high correlation between the independent variables.

• If significant correlation exists between two or more predictors, it becomes challenging to ascertain each predictor's specific impact on the response variable.
• Variables mentioned: * Independent/Response Variable Input (Example: Experience)
• Dependent/Observation Variable Output (Example: Salary)

Dr. P V Siva Teja

​

4. Large Sample Size

A substantial number of samples is needed to obtain accurate estimations of the coefficients. Small datasets can lead to unreliable results in logistic regression.

5. No Outliers

• Outliers represent data points that significantly differentiate themselves from the remainder of the data.
• They may profoundly affect the analysis findings and the accuracy of the model.
• Action: If outliers exist, they must be located and either eliminated or researched further to understand their cause.

Dr. P V Siva Teja

Linear Regression Overview

• Definition: It is a statistical model that analyzes the linear relationship between a dependent variable and a given set of independent variables.
• Linear Relationship: This means that when the value of one or more independent variables changes (increases or decreases), the value of the dependent variable will also change accordingly (increase or decrease).
• The Algorithm: The algorithm functions by finding the best-fit straight line relationship (linear equation) between the two variables.

Practical Example: Salary.csv

• Independent Variable: Years of experience.
• Dependent Variable: Salary of the person.
• Core Principle: The value of the dependent variable (Salary) is predicted using the value of an independent variable (Experience).

Dr. P V Siva Teja

Dr. P V Siva Teja

Types of Linear Regression

• Simple Linear Regression
• Multiple Linear Regression

1. Simple Linear Regression

It is used when we want to predict a target value (Dependent Variable) using only one input feature (Independent Variable). It assumes a straight-line relationship between the two.

Formula:

y = β₀ + β₁x

Where:�y : Predicted value�x : Independent variable�β₀ : Intercept (value of y when x = 0)�β₁ : Slope or coefficient

​

Dr. P V Siva Teja

2. Multiple Linear Regression

It involves more than one independent variable and one dependent variable. The equation for multiple linear regression is:

Formula:

y = β₀ + β₁x₁ + β₂x₂ + … + βₙxₙ

Where:�y : Predicted / Dependent value�x₁, x₂, …, xₙ : Independent variables�β₀ : Intercept�β₁, β₂, …, βₙ : Coefficients corresponding to the predictors

Dr. P V Siva Teja

Best Fit Line in Linear Regression

• In Linear Regression, the best-fit line is the straight line that most accurately represents the relationship between the independent variable (i/p) and the dependent variable (o/p).

Key Characteristics:

• It is the line that minimizes the difference between the actual data points and the predicted values generated by the model.

Dr. P V Siva Teja

The best fit line designing following:

1. Goal of the Best-Fit line
2. Equation of the Best-Fit line
3. Minimizing the error
4. Interpretation of the Best-Fit line

​

1. Goal of the Best-Fit line:-

The goal of linear regression is to find a st. line (straight line) that minimizes the error between the observed data points and the predicted value.

This line helps us predict the dependent variable for new unseen data (test data).

�

Dr. P V Siva Teja

2. Equation of the Best-Fit Line

The line is represented by the following equivalent formulas:

y = mx + c

or

y = β₀ + β₁x

y — Predicted value (Dependent value / Outcome)

x — Input value (Independent value / Predictor)

m (or β₁) — Slope of the line (Gradient / Rate of change)

c (or β₀) — Intercept (y=value when x = 0)

Dr. P V Siva Teja

3. Minimizing the Error

To find the best-fit line, a method called Least Squares (MSE) is used.

How the Method Works

• Objective: This method aims to minimize the Sum of Square difference between the actual value (data points) and the predicted values from the line.
• Residuals: These differences between the actual and predicted values are called residuals.

​

Dr. P V Siva Teja

4. Interpretation of the Best-Fit Line

Understanding the components of the line allows you to describe the relationship between variables.

The Slope (m)

• Definition: The slope indicates how much the dependent variable (y) changes with each unit change in the independent variable (x).
• Formula: The slope can be represented as:

m = dy/dx

dy : Change in the dependent variable�dx : Change in the independent variable

​

The Intercept (c)

Definition:�The intercept represents the predicted value of y when x = 0.

Dr. P V Siva Teja

Advantages:

​

• Simple and Efficient: Easy to implement and computationally fast.
• Good for linear data: Performs exceptionally well when the relationship between variables is linear.
• Interpretability: The relationship between input and output is easy to understand through coefficients.
• Overfitting Mitigation: Less prone to overfitting compared to complex models, especially with regularization.
• Extrapolation: Can be used to predict values outside the range of training data.

Dr. P V Siva Teja

​

Disadvantages:

​

• Linearity assumptions: Assumes a straight-line relationship, which may not exist in real-world data.
• Sensitivity to outliers: A single outlier can significantly skew the regression line.
• Under-fitting: May fail to capture complex patterns in non-linear datasets.
• Multicollinearity: High correlation between independent variables can make the model unstable.
• Assumption of Independence: Assumes that the data points are independent of each other.

Dr. P V Siva Teja

Applications:

​

• Sales Forecasting: Predicting future sales based on historical trends.
• Risk Assessment: Evaluating potential risks in finance or insurance.
• Real Estate price prediction: Estimating property values based on features like size and location.
• Healthcare: Predicting patient outcomes or the spread of diseases.

Dr. P V Siva Teja

Multi-Layer Perceptron’s (MLPs)

In particular, the MLP can solve the XOR problem (or) trivial problems.

• The limitations of a perceptron can be eliminated by stacking multiple perceptrons.
• The result in ANN (Artificial Neural Network) is called a multi-layer perceptron.
• An MLP is composed of TLU's (Threshold Logic Units), called hidden layers, and one final layer of TLU called the output layer, as shown below.

Dr. P V Siva Teja

Dr. P V Siva Teja

The layers close to the input layer are usually called the lower layers, and the ones close to the outputs are usually called the upper layers.

​

Key Characteristics of Artificial Neural Networks (ANN)

• Bias Neurons: Every layer except the output layer (o/p layer) includes a bias neuron and is fully connected to the next layer.
• Deep Neural Networks (DNN): When an ANN contains a deep stack of hidden layers, it is called a Deep Neural Network (DNN).
• Supervised Learning: Multi-Layer Perceptron (MLP) networks are used for the Supervised learning format.
• Training Algorithm: A typical learning algorithm for MLP networks is also called the Backpropagation algorithm.

Dr. P V Siva Teja

Computations Performed on Hidden Layers

The process is divided into two primary steps:

Step 1: Linear Transformation

All inputs are multiplied by their respective weights, and a bias variable is added to the sum. The formula for this linear combination is:

z₁ = x₁w₁ + x₂w₂ + … + xₙwₙ + b

Dr. P V Siva Teja

Step 2: Activation Function

The result of the linear equation (z1) is passed through an activation function.

• The activation function is a non-linear transformation.
• It is applied to the input before sending the signal to the next layer of neurons.

In a Multi-Layer Perceptron (MLP), the system evaluates its performance by calculating the error:

​

error = predicted output − actual output

​

• Minimizing Error: If the error is large, steps are taken to minimize it.
• Backpropagation: For the purpose of error reduction, a process called backpropagation is performed, which adjusts the weights and biases based on the calculated error.

Dr. P V Siva Teja

Backpropagation for Training an MLP

• Definition: It is the process of updating and finding the optimal values of weights or coefficients which helps to minimize the error.
• Purpose: The Backpropagation algorithm is the method for fine-tuning the weights of a neural network with respect to the error rate obtained in the previous iteration or epoch. This is a standard method of training Artificial Neural Networks (ANN).

Dr. P V Siva Teja

Backpropagation Algorithm Overview

There are overall four main steps in the backpropagation algorithm:

• Forward pass
• Error calculation
• Backward pass
• Weights update

1. Forward Pass

This is the first step of the backpropagation process, and it's illustrated below:

• The data (input x1 and x2) is fed to the input layer.
• Then, each input is multiplied by its corresponding weight, and the results are passed to the neurons N1X and N2X of the hidden layers.
• Those neurons apply an activation function to the weighted inputs they receive, and the result passes to the next layer.

​

Dr. P V Siva Teja

Two inputs → x₁, x₂

Two hidden layers → N₁X and N₂X, where X = 1, 2, 3.

Dr. P V Siva Teja

2. Error Calculation (predictions)

• The process continues until the output layer generates the final output (O/P).
• The output of the network is then compared to the ground truth (desired output), and the difference is calculated, resulting in an error value.

Formula: error value = actual output − predicted output

Dr. P V Siva Teja

3. Backward Pass

This is an actual backpropagation step, and cannot be performed without the above forward and error calculation steps. Here is how it works:

• The error value obtained previously is used to calculate the gradient of the loss function.
• The gradient of the error is propagated back through the network, starting from the output layer to the hidden layer.
• The learning rate determines the size of the means update. A smaller learning rate means that the weights are updated by a smaller amount, and vice-versa.

Dr. P V Siva Teja

Dr. P V Siva Teja

4. Weights Update

• The weights are updated in the opposite directions of the gradient, leading to the name "gradient descent." It aims to reduce the error in the next forward pass.
• This process of forward pass, error calculation, backward pass, and weights update continues for multiple epochs until the network performance reaches a satisfactory level or stops improving significantly.

Dr. P V Siva Teja

Advantages

• Easy of implementation
• Programming simplicity
• Flexibility

Disadvantages

• Data quality
• Training duration
• Matrix-based complexity