1 of 64

6CS012 – Artificial Intelligence and Machine Learning��Week – 1�

Introduction to Linear Algebra and Derivatives for Deep Learning��A quick revision on Linear Algebra

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

1

2 of 64

About the module!

What the module is about?

How to pass the module?

2

3 of 64

What the module is about?

Learning Objectives:

  • Apply essential mathematical concepts for understanding Deep Learning algorithms.
  • Explain core principles and architectures of Deep Learning models.
  • Analyze components of learning systems used in Deep Learning design.
  • Implement and evaluate Machine Learning/Deep Learning algorithms.

​

​

3

4 of 64

Learning Outcome:

  • Review and revise some fundamental concepts from mathematics we will be using throughout the course.

​

  • We will discuss:
    • Why do we need Linear Algebra for Machine Learning?
      • Almost Everything you need to know about vector and matrices for Machine Learning.

​

    • A very big picture on Definition of Derivative.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

4

5 of 64

Module Prerequisites

​

    • Introduction to Programming (Level-4)
      • Programming-Python – Recommended.
      • IDE-Jupyter Notebook/Google Colab- Recommended.
    • Concepts and Technologies of AI(Level-5)
    • Computational Mathematics (Level 4)

​

​

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning – Copyright of images not claimed

5

6 of 64

​

​

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning – Copyright of images not claimed

6

7 of 64

Learning Outcomes!!!

  • Review and revise some fundamental concepts from Mathematics – Linear Algebra and Derivative we will be using throughout the course.
    • Cautions !!!
      • We will omit many important topics in Linear algebra and Matrix Calculus, which we believe are not essential for understanding deep learning.
  • In Particular we will discuss:
    • Why do we need Linear Algebra for Machine/Deep Learning?
      • {Almost} Everything we need to know about vector and matrices for Machine/Deep Learning.

​

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

7

8 of 64

Introduction to Linear Algebra

Why to study Linear Algebra for Machine Learning?

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

8

9 of 64

What is Linear Algebra?

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

9

  • Linear algebra is a branch of mathematics focused on vectors, vector spaces, and linear equations.
  • It helps us understand how to work with multiple variables in a structured way.
  • Key Concepts
    • Linear Equations: Relationships between variables, e.g.,

​

    • Vectors and Matrices: Used to represent data and transformations.
    • Transformations: Operations that move or scale data within vector spaces.

​

​

​

10 of 64

Why Linear Algebra for Machine Learning?

  • Representation of Data:
    • In machine learning, data is typically represented as vectors and matrices.

​

    • For example, a dataset might be stored as a matrix where each row is a data point (vector), and each column is a feature.

​

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

10

11 of 64

Why Linear Algebra for Machine Learning?

  • Representation of Data:
    • In machine learning, data is typically represented as vectors and matrices.

​

    • For example, a dataset might be stored as a matrix where each row is a data point (vector), and each column is a feature.

​

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

11

12 of 64

Why Linear Algebra for Machine Learning?

  • Efficient Computing:
    • Matrix operations allow for efficient computations on large datasets.

​

    • Libraries like NumPy, TensorFlow, and PyTorch leverage linear algebra for operations on large matrices and tensors {Vectorizations}, which makes machine learning models faster and more scalable.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

12

13 of 64

Why Linear Algebra for Machine Learning?

  • Understanding {Machine Learning} Algorithms:
    • Training machine learning models often involves solving systems of linear equations.
      • Linear algebra provides the necessary tools to solve these systems efficiently.

​

    • Many machine learning algorithms are based on linear algebra concepts.
      • Linear Regression : Finding the best fitting hyperplane
      • Support Vector Machines (SVM): Dot Product to measure Similarity
      • Neural Network: Matrix Multiplication for forward and backward propagation

​

​

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

13

14 of 64

Summary : Linear Algebra for Machine Learning.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

14

15 of 64

Review: What are Vectors?

15

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning – Copyright of images not claimed

16 of 64

Direction in space

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

16

17 of 64

Point in space�

  • E.g., in 2D we can visualize the data points with respect to a coordinate

​

​

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

17

18 of 64

In Machine Learning

The geometric interpretation of vectors as points in space allow us to consider a training set of input examples in ML as a collection of points in space

So, for eg.

classification can be viewed as discovering how to separate two clusters of points/vectors belonging to different classes

      • Rather than distinguishing images containing cars, planes, buildings etc

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

18

19 of 64

Scalars?

  • A number real or Complex.
  • Scales the vector by the number.

​

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

19

20 of 64

Scalar times vector

20

21 of 64

Scalar times vector

21

22 of 64

Scalar times vector

22

23 of 64

Operations on Vectors

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

23

24 of 64

Addition on Vectors

Add element-wise.

Can only add vectors of equal dimensions.

Associative and commutative

​

24

25 of 64

Element-by-element product (Hadamard product)

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

25

26 of 64

Properties of a vector

  • Let x, y, and z be vectors of the same dimension and let α and β be scalars; then the following properties hold:

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

26

27 of 64

Dot product (inner product)

  • Scalar Product of vectors

27

1 X N

N X 1

1 X 1

Outer dimensions give

size of resulting matrix

28 of 64

Geometric Intuition

​

​

​

​

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

28

Takes 2 vectors of the same dimension, returns a scalar

​

A measure of the “alignment” between two vectors, scaled by the lengths.

29 of 64

Geometric Intuition

Question

What happens when two vectors are orthogonal?

​

Dot product?

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

29

30 of 64

Geometric Intuition

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

30

31 of 64

Vector Space

x + y ∈ V with the addition properties

ax ∈ V with the scalar multiplication properties:

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

31

32 of 64

Linear Combination

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

32

33 of 64

Linear Combination

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

33

34 of 64

Span

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

34

35 of 64

Linear Dependence and Independence

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

35

Linearly dependent

w ∈ span (u, v)

Linearly independent

​

36 of 64

Matrix

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

36

37 of 64

Review What are Matrices?�

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

37

38 of 64

Intuitions on Matrix -1

As a table of data

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

38

39 of 64

Intuitions on Matrix -2

39

40 of 64

Intuitions on Matrix -3

Interpretation 2: Linear Transformation on Vectors

Multiplying a m x n matrix into a n x 1 vector yields a m x 1 vector.

​

We can think of this as a linear function from n-dimensional to m-dimensional space.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

40

Image source : https://mathematica.stackexchange.com/questions/151238/how-to-use-manipulate-to-show-linear-transformation-of-matrix

Ax → y

41 of 64

Intuitions on Matrix -4

Intuition 3: As Linear Equations

​

y = Ax

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

41

42 of 64

Special Matrices.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

42

43 of 64

Special Matrices.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

43

44 of 64

Special Matrices Design Matrix.

  • Structured representation of the input data used for modelling (also known as feature matrix).

​

  • It is a matrix where each row corresponds to a data point (an observation) and each column corresponds to a feature (a variable or predictor) of those data points.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

44

45 of 64

Matrix Operation: Arithmetic.

Matrix – Matrix Sum:

Matrix – Scalar Multiplication:

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

45

  • Operates element-wise
  • Each element is multiplied by the scalar α(alpha)

​

​

​

​

  • Example:

​

​

46 of 64

Matrix Operation: Arithmetic.

Matrix – vector Multiplication:

Computes the dot product of each row of A with x

​

​

Hadamard Product:

    • Multiplies the corresponding elements of A and B

​

​

    • Example:

​

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

46

47 of 64

Matrix Operation: Arithmetic.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

47

Matrix multiplication is an operation that combines two matrices to produce a new matrix. It involves multiplying corresponding elements of rows and columns, then summing the products.

​

A * B = C

Conditions: The number of columns in the first matrix (A) must equal the number of rows in the second matrix (B).

​

​

​

Properties:

Associativity: (A * B) * C = A * (B * C)

Distributivity: A * (B + C) = A * B + A * C

Not Commutative (in general): A * B ≠ B * A

​

48 of 64

Transpose

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

48

49 of 64

Inverse

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

49

50 of 64

Pseudo Inverse of Matrices

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

50

51 of 64

Determinant of a matrix

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

51

52 of 64

Determinant of the matrix

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

52

53 of 64

Rank of a matrix

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

53

54 of 64

Matrix Decomposition-Eigen.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

54

55 of 64

What do matrices do to vectors?

  • The new vector is:

1) rotated

2) scaled

55

(3,5)

(2,1)

56 of 64

Eigen Vector and Eigen Values.

  • Eigenvectors of a square matrix A are non-zero vectors whose direction does not change when A is applied (it only reverses if λ < 0).
  • They are only scaled, by the eigenvalue λ.

Ax = λx

x is an eigenvector of A corresponding to eigenvalue, λ.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

56

​

​

57 of 64

Computing Eigen Vector and Eigen Values.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

57

58 of 64

Eigen Decomposition.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

58

For a matrix A with n linearly independent eigenvectors: A = VΛV⁻¹

V = matrix of eigenvectors

Λ = diagonal matrix of eigenvalues

​

Key Applications of Eigen Decompositions

• Finding matrix inverse: A⁻¹ = VΛ⁻¹V⁻¹ (when no eigenvalue is zero)

• Computing determinant: det(A) = product of eigenvalues

• Analysing matrix properties (positive/negative definite semi-definite, etc.)

​

59 of 64

Eigen Decomposition-Geometric Interpretation.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

59

60 of 64

Review : What are Tensors?

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

60

61 of 64

What are Tensors?

  • A tensor is a multidimensional array and a generalization of the concepts of a vector and a matrix.

​

​

​

​

  • Tensors can have many axes, here is a tensor with three axes:

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

61

62 of 64

Tensor Example.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

62

63 of 64

Plan

  • Tutorial:
    • Review of Derivative for Deep Learning.
    • Some Hands-on Exercise on Vector, Matrices.
  • Workshop – Implement PCA with eigenvalue decomposition for Image Compression Application.

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

63

64 of 64

Thank You!!!

Week - 1 - Introduction to Linear Algebra and Derivatives for Deep Learning

64