Linear Algebra
(BMG5DSE1A3)
Lecture 1
Lecture 1
Systems of Linear Equations
In this lecture, we will introduce linear systems and the method of row reduction to solve them. We will introduce matrices as a convenient structure to represent and solve linear systems. Lastly, we will discuss geometric interpretations of the solution set of a linear system in 2- and 3-dimensions.
1.1 What is a system of linear equations?
Definition 1.1: A system of m linear equations in n unknown variables x1, x2, . . . , xn
is a collection of m equations of the form
a11x1 a21x1 a31x1
+
+
+
a12x2 a22x2 a32x2
+
+
+
a13x3 a23x3 a33x3
+ · · · +
+ · · · +
+ · · · +
a1nxn a2nxn a3nxn
= b1
= b2
= b3
.
.
.
.
.
am1x1
.
+
am2x2
.
+
am3x3
. .
+ · · · +
amnxn
.
= bm
(1.1)
The numbers aij are called the coefficients of the linear system; because there are m equa- tions and n unknown variables there are thefore m × n coefficients. The main problem with a linear system is of course to solve it:
Problem: Find a list of n numbers (s1, s2, . . . , sn) that satisfy the system of linear equa- tions (1.1).
In other words, if we substitute the list of numbers (s1, s2, . . . , sn) for the unknown variables (x1, x2, . . . , xn) in equation (1.1) then the left-hand side of the ith equation will equal bi. We call such a list (s1, s2, . . . , sn) a solution to the system of equations. Notice that we say “a solution” because there may be more than one. The set of all solutions to a linear system is called its solution set. As an example of a linear system, below is a linear
1
Systems of Linear Equations
system consisting of m = 2 equations and n = 3 unknowns:
x1 − 5x2 − 7x3 = 0 5x2 + 11x3 = 1
Here is a linear system consisting of m = 3 equations and n = 2 unknowns:
−5x1 + x2 = −1
πx1 −√5x2 = 0
63x1 − 2x2 = −7
And finally, below is a linear system consisting of m = 4 equations and n = 6 unknowns:
−5x1 + x3 − 44x4 − 55x6 = −1
πx1 − 5x2 − x3 + 4x4 − 5x5 + √5x6 = 0
√
1 2
5
3
4
5
1 1
33
6
63x − 2x − x + ln(3)x + 4x − x = 0
√
63x − 2x
1 2
1
5
— x
3
1
8
4 6
— x − 5x = 5
Example 1.2. Verify that (1, 2, −4) is a solution to the system of equations
2x1 + 2x2 + x3 = 2
x1 + 3x2 − x3 = 11.
Is (1, −1, 2) a solution to the system?
Solution. The number of equations is m = 2 and the number of unknowns is n = 3. There are m × n = 6 coefficients: a11 = 2, a12 = 1, a13 = 1, a21 = 1, a22 = 3, and a23 = −1. And b1 = 0 and b2 = 11. The list of numbers (1, 2, −4) is a solution because
2 · (1) + 2(2) + (−4) = 2
(1) + 3 · (2) − (−4) = 11
On the other hand, for (1, −1, 2) we have that
2(1) + 2(−1) + (2) = 2
but
1 + 3(−1) − 2 = −4 /= 11.
Thus, (1, −1, 2) is not a solution to the system.
A linear system may not have a solution at all. If this is the case, we say that the linear
system is inconsistent:
2
Lecture 1
INCONSISTENT ⇔ NO SOLUTION
A linear system is called consistent if it has at least one solution:
CONSISTENT ⇔ AT LEAST ONE SOLUTION
We will see shortly that a consistent linear system will have either just one solution or infinitely many solutions. For example, a linear system cannot have just 4 or 5 solutions. If it has multiple solutions, then it will have infinitely many solutions.
Example 1.3. Show that the linear system does not have a solution.
−x1 + x2 = 3
x1 − x2 = 1.
Solution. If we add the two equations we get
0 = 4
which is a contradiction. Therefore, there does not exist a list (s1, s2) that satisfies the system because this would lead to the contradiction 0 = 4.
Example 1.4. Let t be an arbitrary real number and let
1
3
2
s = − − 2t
s2 = 3 + t
2
s3 = t.
Show that for any choice of the parameter t, the list (s1, s2, s3) is a solution to the linear system
x1 + x2 + x3 = 0
x1 + 3x2 − x3 = 3.
Solution. Substitute the list (s1, s2, s3) into the left-hand-side of the first equation
3
2
3
2
− − 2t + + t + t = 0
and in the second equation
3
3
3
9
2 2 2 2
− − 2t + 3( + t) − t = − + = 3
Both equations are satisfied for any value of t. Because we can vary t arbitrarily, we get an infinite number of solutions parameterized by t. For example, compute the list (s1, s2, s3)
for t = 3 and confirm that the resulting list is a solution to the linear system.
3
Systems of Linear Equations
1.2 Matrices
We will use matrices to develop systematic methods to solve linear systems and to study the properties of the solution set of a linear system. Informally speaking, a matrix is an array or table consisting of rows and columns. For example,
5x3 − x4 = 7.
4
−2 1
A = 0 2 −8 8
1 0
−4 7 11 −5
is a matrix having m = 3 rows and n = 4 columns. In general, a matrix with m rows and
n columns is a m × n matrix and the set of all such matrices will be denoted by Mm×n.
Hence, A above is a 3 × 4 matrix. The entry of A in the ith row and jth column will be denoted by aij . A matrix containing only one column is called a column vector and a matrix containing only one row is called a row vector. For example, here is a row vector
u = 1 −3
v =
4
and here is a column vector
3
−1
.
We can associate to a linear system three matrices: (1) the coefficient matrix, (2) the output column vector, and (3) the augmented matrix. For example, for the linear system
5x1 − 3x2 + 8x3 = −1
x1 + 4x2 − 6x3 = 0 2x2 + 4x3 = 3
the coefficient matrix A, the output vector b, and the augmented matrix [A b] are:
A = 1 4 −
6 , b = 0 ,
5 8 −1 5
0 2 4 3 0 2 4 3
−3 −3 8 −1
[A b] = 1 4 −6 0 .
we can write this as A ∈ Mm×n.
If a linear system has m equations and n unknowns then the coefficient matrix A must be a
m × n matrix, that is, A has m rows and n columns. Using our previously defined notation,
If we are given an augmented matrix, we can write down the associated linear system in an obvious way. For example, the linear system associated to the augmented matrix
1
0 1 −7 2 −4
0 0 5 −1 7
4 −2 8 12
is
x1 + 4x2 − 2x3 + 8x4 = 12
x2 − 7x3 + 2x4 = −4
Lecture 1
We can study matrices without interpreting them as coefficient matrices or augmented ma- trices associated to a linear system. Matrix algebra is a fascinating subject with numerous applications in every branch of engineering, medicine, statistics, mathematics, finance, biol- ogy, chemistry, etc.
1.3 Solving linear systems
In algebra, you learned to solve equations by first “simplifying” them using operations that do not alter the solution set. For example, to solve 2x = 8 − 2x we can add to both sides
2x and obtain 4x = 8 and then multiply both sides by 1 yielding x = 2. We can do
4
similar operations on a linear system. There are three basic operations, called elementary
operations, that can be performed:
These operations do not alter the solution set. The idea is to apply these operations itera- tively to simplify the linear system to a point where one can easily write down the solution set. It is convenient to apply elementary operations on the augmented matrix [A b] repre- senting the linear system. In this case, we call the operations elementary row operations, and the process of simplifying the linear system using these operations is called row reduc- tion. The goal with row reducing is to transform the original linear system into one having a triangular structure and then perform back substitution to solve the system. This is best explained via an example.
Example 1.5. Use back substitution on the augmented matrix
1
0 1 −1 0
0 −2 −4
0 0 1 1
to solve the associated linear system.
Solution. Notice that the augmented matrix has a triangular structure. The third row corresponds to the equation x3 = 1. The second row corresponds to the equation
x2 − x3 = 0
and therefore x2 = x3 = 1. The first row corresponds to the equation
x1 − 2x3 = −4
and therefore
x1 = −4 + 2x3 = −4 + 2 = −2.
Therefore, the solution is (−2, 1, 1).
5
Systems of Linear Equations
Example 1.6. Solve the linear system using elementary row operations.
−3x1 + 2x2 + 4x3 = 12
x1 − 2x3 = −4 2x1 − 3x2 + 4x3 = −3
Solution. Our goal is to perform elementary row operations to obtain a triangular structure and then use back substitution to solve. The augmented matrix is
−3
2 4 12
1 0 −2 −4 .
2 −3 4 −3
Interchange Row 1 (R1) and Row 2 (R2):
1 0 −2 −4
−3 2 4 12
R ↔R
1 2
−−−−→
1
−3 2 4 12
0 −2 −4
2 −3 4 −3 2 −3 4 −3
As you will see, this first operation will simplify the next step. Add 3R1 to R2:
0 −2 −4
−3 2 4 12
2 −3 4 −3
3R1 +R2
1 1
−−−−→ 0
0 −2 −4
2 −2 0
2 −3 4 −3
Add −2R1 to R3:
1
0 −2 −4
0 2 −2 0
2 −3 4 −3
−2R +R
1 3
1
−−−−−→ 0
0 −2 −4
2 −2 0
0 −3 8 5
2
Multiply R2 by 1 :
2
1 1
R2
Add 3R2 to R3:
1
1 0 −2 −4 0 −2 −4
0 2 −2 0 −−→ 0 1 −1 0
0 −3 8 5 0 −3 8 5
0 −2 −4
0 1 −1 0
0 −3 8 5
3R2 +R3
1
−−−−→ 0
0 −2 −4
1 −1 0
0 0 5 5
5
Multiply R3 by 1 :
5
So now use back substitution to solve. The linear system associated to the row reduced
6
1 0 −2 −4
0 1 −1 0 −−→ 0
0 0 5 5 0
1 1 0 −2 −4
R3
1 −1 0
0 1 1
We can continue row reducing but the row reduced augmented matrix is in triangular form.
Lecture 1
augmented matrix is
x1 − 2x3 = −4
x2 − x3 = 0
x3 = 1
The last equation gives that x3 = 1. From the second equation we obtain that x2 − x3 = 0, and thus x2 = 1. The first equation then gives that x1 = −4 + 2(1) = −2. Thus, the solution to the original system is (−2, 1, 1). You should verify that (−2, 1, 1) is a solution to the original system.
The original augmented matrix of the previous example is
0 0 0 −1
7
M = 1 0 −2 −4
−3 2 4 12
→
2 −3 4 −3
After row reducing we obtained the row reduced matrix
−3x1 + 2x2 + 4x3 = 12
x1 − 2x3 = −4
2x1 − 3x2 + 4x3 = −3.
1
N = 0 1 −1 0
0 0 1 1
0 −2 −4
→
x1 − 2x3 = −4
x2 − x3 = 0
x3 = 1.
Although the two augmented matrices M and N are clearly distinct, it is a fact that they
have the same solution set.
Example 1.7. Using elementary row operations, show that the linear system is inconsistent.
x1 + 2x3 = 1 x2 + x3 = 0 2x1 + 4x3 = 1
Solution. The augmented matrix is
0 2
0 1 1 0
2 0 4 1
1 1
Perform the operation −2R1 + R3:
1
0 2
1
0 1 1 0
2 0 4 1
−2R +R
1 3
1
0 2
−−−−−→ 0 1 1 0
0 0 0 −1
1
The last row of the simplified augmented matrix
1
0
0 2
1 1
1
0
8
Systems of Linear Equations
corresponds to the equation
0x1 + 0x2 + 0x3 = −1
Obviously, there are no numbers x1, x2, x3 that satisfy this equation, and therefore, the linear system is inconsistent, i.e., it has no solution. In general, if we obtain a row in an augmented matrix of the form
0 0 0 · · · 0 c
where c is a nonzero number, then the linear system is inconsistent. We will call this type of row an inconsistent row. However, a row of the form
0 1 0 0 0
corresponds to the equation x2 = 0 which is perfectly valid.
1.4 Geometric interpretation of the solution set
The set of points (x1, x2) that satisfy the linear system
x1 − 2x2 = −1
−x1 + 3x2 = 3
(1.2)
is the intersection of the two lines determined by the equations of the system. The solution for this system is (3, 2). The two lines intersect at the point (x1, x2) = (3, 2), see Figure 1.1.
Figure 1.1: The intersection point of the two lines is the solution of the linear system (1.2)
Similarly, the solution of the linear system
x1 − 2x2 + x3 = 0 2x2 − 8x3 = 8
−4x1 + 5x2 + 9x3 = −9
(1.3)
9
Lecture 1
is the intersection of the three planes determined by the equations of the system. In this case, there is only one solution: (29, 16, 3). In the case of a consistent system of two equations, the solution set is the line of intersection of the two planes determined by the equations of the system, see Figure 1.2.
the solution set is this line
x1 − 2x2 + x3 = 0
−4x1 + 5x2 + 9x3 = −9
Figure 1.2: The intersection of the two planes is the solution set of the linear system (1.3)
After this lecture you should know the following:
10
Systems of Linear Equations
Lecture 2
11
Lecture 2
Row Reduction and Echelon Forms
In this lecture, we will get more practice with row reduction and in the process introduce two important types of matrix forms. We will also discuss when a linear system has a unique solution, infinitely many solutions, or no solution. Lastly, we will introduce a convenient parameter called the rank of a matrix.
2.1 Row echelon form (REF)
Consider the linear system
x1 + 5x2 − 2x4 − x5 + 7x6 = −4
2x2 − 2x3 + 3x6 = 0
−9x4 − x5 + x6 = −1
5x5 + x6 = 5
0 = 0
having augmented matrix
1
5 0 −2 −1 7 −4
0 2 −2 0 0 3 0
0 0 0 −9 −1 1 −1 .
0 0 0 0 5 1 5
0 0 0 0 0 0 0
The above augmented matrix has the following properties:
P1. All nonzero rows are above any rows of all zeros.
P2. The leftmost nonzero entry of a row is to the right of the leftmost nonzero entry of the row above it.
12
Row Reduction and Echelon Forms
Any matrix satisfying properties P1 and P2 is said to be in row echelon form (REF). In REF, the leftmost nonzero entry in a row is called a leading entry:
1 5 0 −2 −1 7 −4
0 2 −2 0 0 3 0
0 0 0 −9 −1 1 −1
0 0 0 0 5 1 5
0 0 0 0 0 0 0
A consequence of property P2 is that every entry below a leading entry is zero:
1 5 0 −2 −4 −1 −7
0 2 −2 0 0 3 0
0 0 0 −9 −1 1 −1
0 0 0 0 5 1 5
0 0 0 0 0 0 0
We can perform elementary row operations, or row reduction, to transform a matrix into REF.
Example 2.1. Explain why the following matrices are not in REF. Use elementary row operations to put them in REF.
M = 0 0
3 3 7
N = 0 3 −1 1
5 0 −3
0 0 0 6 −5 2
Solution. Matrix M fails property P1. To put M in REF we interchange R2 with R3:
−1 0 | 0 0 |
1 | 3 |
−1 | 0 0 |
1 | 3 |
R2 ↔R3
−1 0
M = 0 0 0 −−−−→ 0
1 3 0
0 0 0 0 0 0
3 3 3 3
The matrix N fails property P2. To put N in REF we perform the operation −2R2 + R3 →
R3:
0 6 −5 2
−2R +R
2 3
7 7
0 3 −1 1 −−−−−→ 0 3 −1 1
0 0 −3 0
5 0 −3 5 0 −3
Why is REF useful? Certain properties of a matrix can be easily deduced if it is in REF. For now, REF is useful to us for solving a linear system of equations. If an augmented matrix is in REF, we can use back substitution to solve the system, just as we did in Lecture 1. For example, consider the system
8x1 − 2x2 + x3 = 4
3x2 − x3 = 7
2x3 = 4
13
Lecture 2
whose augmented matrix is already in REF:
8
−2 1
4
0 3 −1 7
0 0 2 4
From the last equation we obtain that 2x3 = 4, and thus x3 = 2. Substituting x3 = 2 into the second equation we obtain that x2 = 3. Substituting x3 = 2 and x2 = 3 into the first equation we obtain that x1 = 1.
2.2 Reduced row echelon form (RREF)
Although REF simplifies the problem of solving a linear system, later on in the course we will need to completely row reduce matrices into what is called reduced row echelon form (RREF). A matrix is in RREF if it is in REF (so it satisfies properties P1 and P2) and in addition satisfies the following properties:
P3. The leading entry in each nonzero row is a 1.
P4. All the entries above (and below) a leading 1 are all zero.
A leading 1 in the RREF of a matrix is called a pivot. For example, the following matrix in RREF:
6 0 3 0
0 0 1 −4 0 5
0 0 0 0 1 7
1 0
has three pivots:
1
6 0 3 0
0 0 1 −4 0 5
0 0 0 0 1 7
0
Example 2.2. Use row reduction to transform the matrix into RREF.
3 −6 6 4 −5
3 −7 8 −5 8 9
0
3 −9 12 −9 6 15
Solution. The first step is to make the top leftmost entry nonzero:
3 −9 12 −9 6 15
R3 ↔R1
0 3
3 −6 6 4 −5 −9
3 −7 8 −5 8 9
−−−−→ 3
12 −9 6 15
−7 8 −5 8 9
0 3 −6 6 4 −5
Now create a leading 1 in the first row:
3
3
−9 12 −9 6 15 1 R1
3 −7 8 −5 8 9
0 3 −6 6 4 −5
1
−3 4 −3 2
−−→ 3 −7 8 −5 8 9
0 3 −6 6 4 −5
5
14
Row Reduction and Echelon Forms
Create zeros under the newly created leading 1:
−3 4 −3 2
3 −7 8 −5 8 9
0 3 −6 6 4 −5
1 5
−3R1 +R2
1
−−−−−→ 0
−3 4 −3 2
2 −4
0 3 −6
5
4 2 −6
6 4 −5
Create a leading 1 in the second row:
1
−3 4 −3 2
5
0 2 −4 4 2 −6
0 3 −6 6 4 −5
2
1
R2
1
−3 4 −3 2
−−→ 0 1 −2
0 3 −6
5
2 1 −3
6 4 −5
Create zeros under the newly created leading 1:
−3 4 −3 2
1 5
−3R2 +R3
1
−3 4 −3 2
0 1 −2 2 1 −3 −−−−−→ 0
0 3 −6 6 4 −5 0 0 0 0
5
1 −2 2 1 −3
1 4
We have now completed the top-to-bottom phase of the row reduction algorithm. In the next phase, we work bottom-to-top and create zeros above the leading 1’s. Create zeros above the leading 1 in the third row:
−3 4 −3 2
0 1 −2 2 1 −3
0 0 0 0 1 4
1 5
−R3 +R2
1
−3 4 −3 2
−−−−−→ 0
5
1
−3 4 −3 2
0 1 −2 2 0 −7
0 0 0 0 1 4
5
−2R3 +R1
1
1 −2 2 0 −7
0 0 0 0 1 4
−3 4
−−−−−→ 0
1 −2 2 0 −7
0 0 0 0 1 4
−3 0 −3
Create zeros above the leading 1 in the second row:
1
0 1 −2 2 0 −7
−3 4 −3 0 −3
3R +R
2 1
1
−−−−→ 0
0 −2 3 0 −24
1 −2 2 0 −7
0 0 0 0 1 4 0 0 0 0 1 4
This completes the row reduction algorithm and the matrix is in RREF.
Example 2.3. Use row reduction to solve the linear system.
2x1 + 4x2 + 6x3 = 8 x1 + 2x2 + 4x3 = 8 3x1 + 6x2 + 9x3 = 12
Solution. The augmented matrix is
2
4 6
1 2 4 8
3 6 9 12
8
15
Lecture 2
Create a leading 1 in the first row:
2
8
2
1 R1
1
4 6 2 3
1 2 4 8 −−→ 1 2 4 8
3 6 9 12 3 6 9 12
4
Create zeros under the first leading 1:
−R1 +R2
1
1 2 3 4 2 3
1 2 4 8 −−−−−→ 0 0 1 4
3 6 9 12 3 6 9 12
4
1
4
−3R +R
1 3
1
2 3 2 3
0 0 1 4 −−−−−→ 0 0 1 4
3 6 9 12 0 0 0 0
4
The system is consistent, however, there are only 2 nonzero rows but 3 unknown variables. This means that the solution set will contain 3 − 2 = 1 free parameter. The second row in the augmented matrix is equivalent to the equation:
x3 = 4.
The first row is equivalent to the equation:
x1 + 2x2 + 3x3 = 4
and after substituting x3 = 4 we obtain
x1 + 2x2 = −8.
We now must choose one of the variables x1 or x2 to be a parameter, say t, and solve for the remaining variable. If we set x2 = t then from x1 + 2x2 = −8 we obtain that
x1 = −8 − 2t.
We can therefore write the solution set for the linear system as
x1 = −8 − 2t
x2 = t x3 = 4
(2.1)
2
where t can be any real number. If we had chosen x1 to be the parameter, say x1 = t, then the solution set can be written as
x1 = t
1
2
x = −4 − t
(2.2)
x3 = 4
Although (2.1) and (2.2) are two different parameterizations, they both give the same solution set.
16
Row Reduction and Echelon Forms
In general, if a linear system has n unknown variables and the row reduced augmented matrix has r leading entries, then the number of free parameters d in the solution set is
d = n − r.
Thus, when performing back substitution, we will have to set d of the unknown variables to arbitrary parameters. In the previous example, there are n = 3 unknown variables and the row reduced augmented matrix contained r = 2 leading entries. The number of free parameters was therefore
d = n − r = 3 − 2 = 1.
Because the number of leading entries r in the row reduced coefficient matrix determine the number of free parameters, we will refer to r as the rank of the coefficient matrix:
r = rank(A).
Later in the course, we will give a more geometric interpretation to rank(A).
Example 2.4. Solve the linear system represented by the augmented matrix
1
0 1 −3 3 1 −5
−7 2 −5 8 10
0 0 0 1 −1 4
Solution. The number of unknowns is n = 5 and the augmented matrix has rank r = 3 (leading entries). Thus, the solution set is parameterized by d = 5 − 3 = 2 free variables, call them t and s. The last equation of the augmented matrix is x4 − x5 = 4. We choose x5 to be the first parameter so we set x5 = t. Therefore, x4 = 4 + t. The second equation of the augmented matrix is
x2 − 3x3 + 3x4 + x5 = −5
and the unassigned variables are x2 and x3. We choose x3 to be the second parameter, say
x3 = s. Then
x2 = −5 + 3x3 − 3x4 − x5
= −5 + 3s − 3(4 + t) − t
= −17 − 4t + 3s.
We now use the first equation of the augmented matrix to write x1 in terms of the other variables:
x1 = 10 + 7x2 − 2x3 + 5x4 − 8x5
= 10 + 7(−17 − 4t + 3s) − 2s + 5(4 + t) − 8t
= −89 − 31t + 19s
17
Lecture 2
Thus, the solution set is
x1 = −89 − 31t + 19s x2 = −17 − 4t + 3s
x3 = s
x4 = 4 + t x5 = t
where t and s are arbitrary real numbers.. Choose arbitrary numbers for t and s and substitute the corresponding list (x1, x2, . . . , x5) into the system of equations to verify that it is a solution.
The REF or RREF of an augmented matrix leads to three distinct possibilities for the solution set of a linear system.
Theorem 2.5: Let [A b] be the augmented matrix of a linear system. One of the following distinct possibilities will occur:
In Case 1., the linear system is inconsistent and thus has no solution. In Case 2., the linear system is consistent and has only one (and thus unique) solution. This case occurs when r = rank(A) = n since then the number of free parameters is d = n − r = 0. In Case 3., the linear system is consistent and has infinitely many solutions. This case occurs when r < n and thus d = n − r > 0 is the number of free parameters.
After this lecture you should know the following:
18
Row Reduction and Echelon Forms
Lecture 3
19
Lecture 3
Vector Equations
In this lecture, we introduce vectors and vector equations. Specifically, we introduce the linear combination problem which simply asks whether it is possible to express one vector in terms of other vectors; we will be more precise in what follows. As we will see, solving the linear combination problem reduces to solving a linear system of equations.
3.1 Vectors in Rn
Recall that a column vector in Rn is a n × 1 matrix. From now on, we will drop the “column” descriptor and simply use the word vectors. It is important to emphasize that a vector in Rn is simply a list of n numbers; you are safe (and highly encouraged!) to forget the idea that a vector is an object with an arrow. Here is a vector in R2:
v =
3
−1
.
Here is a vector in R3:
−3
11
v = 0 .
Here is a vector in R6:
9
0
6
0
v = −3 .
3
To indicate that v is a vector in Rn, we will use the notation v ∈ Rn. The mathematical
symbol ∈ means “is an element of”. When we write vectors within a paragraph, we will write
them using list notation instead of column notation, e.g., v = (−1, 4) instead of v =
4
−1 .
20
Vector Equations
We can add/subtract vectors, and multiply vectors by numbers or scalars. For example, here is the addition of two vectors:
0 4 4
9
0
2 1
+ =
−5 −3 −8
9
3
.
And the multiplication of a scalar with a vector:
1 3
3 −3 = −9 .
5 15
And here are both operations combined:
4 −2 −8 −6 −14
−2 −8 + 3 9 = 16 + 27 = 43 .
3 4 −6 12 6
These operations constitute “the algebra” of vectors. As the following example illustrates, vectors can be used in a natural way to represent the solution of a linear system.
Example 3.1. Write the general solution in vector form of the linear system represented by the augmented matrix
A b = 0
1 −3 3 1 −5
1 −7 2 −5 8 10
x =
3
x
4
x5
x =
x −17 − 4t + 3t
2 1 2
t2
4 + t1
t1
4
0
= 0 + t
1
1
1
2
−17 −4 3
0 + t 1
0
0
0 0 0 1 −1 4
Solution. The number of unknowns is n = 5 and the associated coefficient matrix A has rank r = 3. Thus, the solution set is parametrized by d = n − r = 2 parameters. This system was considered in Example 2.4 and the general solution was found to be
x1 = −89 − 31t1 + 19t2
x2 = −17 − 4t1 + 3t2
x3 = t2
x4 = 4 + t1
x5 = t1
where t1 and t2 are arbitrary real numbers. The solution in vector form therefore takes the form
x1 −89 − 31t1 + 19t2 −89 −31 19
21
Lecture 3
A fundamental problem in linear algebra is solving vector equations for an unknown vector. As an example, suppose that you are given the vectors
1
v = −
4 −2
2
−14
8 , v = 9 , b = 43 ,
3 4 6
and asked to find numbers x1 and x2 such that x1v1 + x2v2 = b, that is,
1
x −
4
2
−2 −14
8 + x 9 = 43 .
3 4 6
Here the unknowns are the scalars x1 and x2. After some guess and check, we find that
x1 = −2 and x2 = 3 is a solution to the problem since
4 −2 −14
−2 −8 + 3 9 = 43 .
3 4 6
In some sense, the vector b is a combination of the vectors v1 and v2. This motivates the following definition.
Definition 3.2: Let v1, v2, . . . , vp be vectors in Rn. A vector b is said to be a linear combination of the vectors v1, v2, . . . , vp if there exists scalars x1, x2, . . . , xp such that x1v1 + x2v2 + · · · + xpvp = b.
The scalars in a linear combination are called the coefficients of the linear combination. As an example, given the vectors
1
v = −
3
−6
2 , v = 4 , v
2 3
−3
= 5 , b = 0
6 −27
1 −2 −1
you can verify (and you should!) that
3v1 + 4v2 − 2v3 = b.
Therefore, we can say that b is a linear combination of v1, v2, v3 with coefficients x1 = 3,
x2 = 4, and x3 = −2.
3.2 The linear combination problem
The linear combination problem is the following:
22
Vector Equations
Problem: Given vectors v1, . . . , vp and b, is b a linear combination of v1, v2, . . . , vp? For example, say you are given the vectors
1
1
v = 2 , v
1 2
1
0
= 1 , v
3
= 1
2
2
and also
0
b = 1 .
−2
Does there exist scalars x1, x2, x3 such that
x1v1 + x2v2 + x3v3 = b? (3.1)
For obvious reasons, equation (3.1) is called a vector equation and the unknowns are x1, x2, and x3. To gain some intuition with the linear combination problem, let’s do an example by inspection.
Example 3.3. Let v1 = (1, 0, 0), let v2 = (0, 0, 1), let b1 = (0, 2, 0), and let b2 = (−3, 0, 7). Are b1 and b2 linear combinations of v1, v2?
Solution. For any scalars x1 and x2
x1 0 x1 0
1 1 2 2
x v + x v = 0 + 0 = 0 =
/ 2
0 x2 x2 0
and thus no, b1 is not a linear combination of v1, v2, v3. On the other hand, by inspection we have that
−3 0 −3
−3v + 7v = 0 + 0 = 0 = b
1 2 2
0 7 7
and thus yes, b2 is a linear combination of v1, v2, v3. These examples, of low dimension, were more-or-less obvious. Going forward, we are going to need a systematic way to solve the linear combination problem that does not rely on pure inspection.
We now describe how the linear combination problem is connected to the problem of solving a system of linear equations. Consider again the vectors
1 1 2 0
v = 2 , v
1 2
1 0
= 1 , v
3
= 1 , b = 1 .
2 −2
Does there exist scalars x1, x2, x3 such that
x1v1 + x2v2 + x3v3 = b?
(3.2)
23
Lecture 3
1 1 2 2 3 3 1 2 3 1 2 3
First, let’s expand the left-hand side of equation (3.2):
x1 x2 2x3 x1 + x2 + 2x3
x v + x v + x v = 2x + x + x = 2x + x + x .
1 2 3
x1 0 2x3 x1 + 2x3
We want equation (3.2) to hold so let’s equate the expansion x1v1 + x2v2 + x3v3 with b. In other words, set
x1 + x2 + 2x3 0
2x + x + x = 1 .
x1 + 2x3 −2
Comparing component-by-component in the above relationship, we seek scalars x1, x2, x3
satisfying the equations
x1 + x2 + 2x3 = 0
2x1 + x2 + x3 = 1 (3.3)
x1 + 2x3 = −2.
This is just a linear system consisting of m = 3 equations and n = 3 unknowns! Thus, the linear combination problem can be solved by solving a system of linear equations for the unknown scalars x1, x2, x3. We know how to do this. In this case, the augmented matrix of the linear system (3.3) is
1 2
[A b] = 2 1 1 1
1 0 2 −2
1 0
Notice that the 1st column of A is just v1, the second column is v2, and the third column is v3, in other words, the augment matrix is
1
[A b] = v v2 v3
b
Applying the row reduction algorithm, the solution is
x1 = 0, x2 = 2, x3 = −1
and thus these coefficients solve the linear combination problem. In other words,
0v1 + 2v2 − v3 = b
In this case, there is only one solution to the linear system, so b can be written as a linear combination of v1, v2, . . . , vp in only one (or unique) way. You should verify these computations.
We summarize the previous discussion with the following:
The problem of determining if a given vector b is a linear combination of the vectors
v1, v2, . . . , vp is equivalent to solving the linear system of equations with augmented matrix
A
b = v1 v2 · · · vp b .
24
Vector Equations
Applying the existence and uniqueness Theorem 2.5, the only three possibilities to the linear combination problem are:
can be written as a linear combination of v1, v2, . . . , vp in infinitely many ways.
Example 3.4. Is the vector b = (7, 4, −3) a linear combination of the vectors
1 2
1
v = −
−5
2
6
2 , v = 5 ?
Solution. Form the augmented matrix:
1 2
v1 v2 b = −2 5 4
7
−5 6 −3
The RREF of the augmented matrix is
0
0 1 2
0 0 0
1 3
and therefore the solution is x1 = 3 and x2 = 2. Therefore, yes, b is a linear combination of
v1, v2:
1 2
3v + 2v = 3 −
1 2 7
2 + 2 5 = 4 = b
−5 6 −3
Notice that the solution set does not contain any free parameters because n = 2 (unknowns) and r = 2 (rank) and so d = 0. Therefore, the above linear combination is the only way to write b as a linear combination of v1 and v2.
Example 3.5. Is the vector b = (1, 0, 1) a linear combination of the vectors
1 0 2
1
v =
2
2
0
3
0 , v = 1 , v =
4
1 ?
25
Lecture 3
Solution. The augmented matrix of the corresponding linear system is
1 1
0 2
0 1 1 0 .
2 0 4 1
After row reducing we obtain that
1
1
0 2
0 1 1 0 .
0 0 0 −1
The last row is inconsistent, and therefore the linear system does not have a solution. There- fore, no, b is not a linear combination of v1, v2, v3.
Example 3.6. Is the vector b = (8, 8, 12) a linear combination of the vectors
2 4 6
3
1 2
6
v = 1 , v = 2 ,
3
9
v = 4 ?
Solution. The augmented matrix is
2
8
REF
1
4
4 6 2 3
1 2 4 8 −−→ 0 0 1 4 .
3 6 9 12 0 0 0 0
The system is consistent and therefore b is a linear combination of v1, v2, v3. In this case, the solution set contains d = 1 free parameters and therefore, it is possible to write b as a linear combination of v1, v2, v3 in infinitely many ways. In terms of the parameter t, the solution set is
x1 = −8 − 2t
x2 = t x3 = 4
Choosing any t gives scalars that can be used to write b as a linear combination of v1, v2, v3. For example, choosing t = 1 we obtain x1 = −10, x2 = 1, and x3 = 4, and you can verify
that
1 2 3
−10v + v + 4v = −
2 4 6 8
10 1 + 2 + 4 4 = 8 = b
3 6 9 12
Or, choosing t = −2 we obtain x1 = −4, x2 = −2, and x3 = 4, and you can verify that
1 2 3
−4v − 2v + 4v = −
2 4 6 8
3
6
9
12
4 1 − 2 2 + 4 4 = 8 = b
26
Vector Equations
We make a few important observations on linear combinations of vectors. Given vectors v1, v2, . . . , vp, there are certain vectors b that can be written as a linear combination of v1, v2, . . . , vp in an obvious way. The zero vector b = 0 can always be written as a linear combination of v1, v2, . . . , vp:
0 = 0v1 + 0v2 + · · · + 0vp.
Each vi itself can be written as a linear combination of v1, v2, . . . , vp, for example,
v2 = 0v1 + (1)v2 + 0v3 + · · · + 0vp.
More generally, any scalar multiple of vi can be written as a linear combination of v1, v2, . . . , vp, for example,
xv2 = 0v1 + xv2 + 0v3 + · · · + 0vp.
By varying the coefficients x1, x2, . . . , xp, we see that there are infinitely many vectors b that can be written as a linear combination of v1, v2, . . . , vp. The “space” of all the possible linear combinations of v1, v2, . . . , vp has a name, which we introduce next.
3.3 The span of a set of vectors
Given a set of vectors {v1, v2, . . . , vp}, we have been considering the problem of whether or not a given vector b is a linear combination of {v1, v2, . . . , vp}. We now take another point of view and instead consider the idea of generating all vectors that are a linear combination of {v1, v2, . . . , vp}. So how do we generate a vector that is guaranteed to be a linear combination of {v1, v2, . . . , vp}? For example, if v1 = (2, 1, 3), v2 = (4, 2, 6) and v3 = (6, 4, 9) then
2 4 6 8
1 2 3
−10v + v + 4v = −
10 1 + 2 + 4 4 = 8 .
3 6 9 12
Thus, by construction, the vector b = (8, 8, 12) is a linear combination of {v1, v2, v3}. This discussion leads us to the following definition.
Definition 3.7: Let v1, v2, . . . , vp be vectors. The set of all vectors that are a linear combination of v1, v2, . . . , vp is called the span of v1, v2, . . . , vp, and we denote it by
S = span{v1, v2, . . . , vp}.
By definition, the span of a set of vectors is a collection of vectors, or a set of vectors. If b is a linear combination of v1, v2, . . . , vp then b is an element of the set span{v1, v2, . . . , vp}, and we write this as
b ∈ span{v1, v2, . . . , vp}.
27
Lecture 3
By definition, writing that b ∈ span{v1, v2, . . . , vp} implies that there exists scalars x1, x2, . . . , xp
such that
x1v1 + x2v2 + · · · + xpvp = b.
Even though span{v1, v2, . . . , vp} is an infinite set of vectors, it is not necessarily true that
it is the whole space Rn.
The set span{v1, v2, . . . , vp} is just a collection of infinitely many vectors but it has some geometric structure. In R2 and R3 we can visualize span{v1, v2, . . . , vp}. In R2, the span of a single nonzero vector, say v ∈ R2, is a line through the origin in the direction of v, see Figure 3.1.
Figure 3.1: The span of a single non-zero vector in R2.
In R2, the span of two vectors v1, v2 ∈ R2 that are not multiples of each other is all of R2. That is, span{v1, v2} = R2. For example, with v1 = (1, 0) and v2 = (0, 1), it is true that span{v1, v2} = R2. In R3, the span of two vectors v1, v2 ∈ R3 that are not multiples of each other is a plane through the origin containing v1 and v2, see Figure 3.2. In R3, the
— 3
— 2
— 4
— 4
— 3
— 1
z 0
— 2
1
2
— 4
— 3
— 1
sspan{v,,w}
3
4
— 2
0
y 1
— 1
0
1 x
2
2
3
3
4
Figure 3.2: The span of two vectors, not multiples of each other, in R3.
span of a single vector is a line through the origin, and the span of three vectors that do not depend on each other (we will make this precise soon) is all of R3.
Example 3.8. Is the vector b = (7, 4, −3) in the span of the vectors v1 = (1, −2, −5), v2 = (2, 5, 6)? In other words, is b ∈ span{v1, v2}?
28
Vector Equations
Solution. By definition, b is in the span of v1 and v2 if there exists scalars x1 and x2 such that
x1v1 + x2v2 = b,
that is, if b can be written as a linear combination of v1 and v2. From our previous discussion
1
on the linear combination problem, we must consider the augmented matrix v v2 b .
Using row reduction, the augmented matrix is consistent and there is only one solution (see Example 3.4). Therefore, yes, b ∈ span{v1, v2} and the linear combination is unique.
Example 3.9. Is the vector b = (1, 0, 1) in the span of the vectors v1 = (1, 0, 2), v2 =
(0, 1, 0), v3 = (2, 1, 4)?
Solution. From Example 3.5, we have that
REF
1
0 2
v1 v2 v3 b −−→ 0 1 1 0
0 0 0 −1
1
The last row is inconsistent and therefore b is not in span{v1, v2, v3}.
Example 3.10. Is the vector b = (8, 8, 12) in the span of the vectors v1 = (2, 1, 3), v2 =
(4, 2, 6), v3 = (6, 4, 9)?
Solution. From Example 3.6, we have that
REF
1
4
2 3
v1 v2 v3 b −−→ 0 0 1 4 .
0 0 0 0
The system is consistent and therefore b ∈ span{v1, v2, v3}. In this case, the solution set contains d = 1 free parameters and therefore, it is possible to write b as a linear combination of v1, v2, v3 in infinitely many ways.
Example 3.11. Answer the following with True or False, and explain your answer.
(a) The vector b = (1, 2, 3) is in the span of the set of vectors
−1 2 4
0 0 0
3 , −7 , −5 .
1
(b) The solution set of the linear system whose augmented matrix is v v2 v3 b is the
same as the solution set of the vector equation x1v1 + x2v2 + x3v3 = b.
1
(c) Suppose that the augmented matrix v v2 v3 b has an inconsistent row. Then
either b can be written as a linear combination of v1, v2, v3 or b ∈ span{v1, v2, v3}.
(d) The span of the vectors {v1, v2, v3} (at least one of which is nonzero) contains only the vectors v1, v2, v3 and the zero vector 0.
29
Lecture 3
After this lecture you should know the following:
and the problem of writing b as a linear combination of v1, v2, . . . , vp
30
Vector Equations
Lecture 4
31
Lecture 4
The Matrix Equation Ax = b
In this lecture, we introduce the operation of matrix-vector multiplication and how it relates to the linear combination problem.
4.1 Matrix-vector multiplication
We begin with the definition of matrix-vector multiplication.
Definition 4.1: Given a matrix A ∈ Mm×n and a vector x ∈ Rn,
A =
a · · ·
a
· · · a
2n
.
.
. . . .
. . .
.
am1 am2 am3 amn
11 a12 a13 a1n
21 a22 a23
, x =
x
2
.
.
· · · xn
x1
,
Ax =
we define the product of A and x as the vector Ax in Rm given by
a
11 a12 a13
21 a22
23
2n
.
.
.
. . .
. . .
.
am1 am2 am3
· · ·
amn
`
˛A¸
xn
x ` ˛x¸ x
2 21 1 22 2
a · · · a x a x + a x + · · · + a x
2n n
. .
. .
am1x1 + am2x2 + · · · + amnxn
a · · · a1n x1 a11x1 + a12x2 + · · · + a1nxn
=
.
For the product Ax to be well-defined, the number of columns of A must equal the number of components of x. Another way of saying this is that the outer dimension of A must equal the inner dimension of x:
(m × n) · (n × 1) → m × 1
Example 4.2. Compute Ax.
32
The Matrix Equation Ax = b
(a)
A = 1
−1 3 0 , x =
2
−4
−3
8
(b)
A =
3 3 −2
4 −4 −1
1
, x = 0
−1
(c)
A =
−1 1 0
4 1 −2
3 −3 3
0 −2 −3
, x = 2
−1
−2
Solution. We compute: (a)
Ax = 1 −1 3 0
2
−4
−3
8
= (1)(2) + (−1)(−4) + (3)(−3) + (0)(8) = −3
(b)
Ax =
3 3 −2
4 −4 −1
1
0
−1
=
(3)(1) + (3)(0) + (−2)(−1)
(4)(1) + (−4)(0) + (−1)(−1)
=
5
5
33
Lecture 4
(c)
Ax =
−1 1
0 −1
4 1 −2
3 −3 3
0 −2 −3
2
−2
=
(−1)(−
1) + (1)(2) + (0)(−2)
(4)(−
1) + (1)(2) + (−
2)(−2)
(3)(−1) + (−3)(2) + (3)(−2)
(0)(−1) + (−2)(2) + (−3)(−2)
=
3
2
−15
2
We now list two important properties of matrix-vector multiplication.
Theorem 4.3: Let A be an m × n a matrix.
A(u + v) = Au + Av.
A(cu) = c(Au).
Example 4.4. For the given data, verify that the properties of Theorem 4.3 hold:
A =
−1 2
3 −3
2 1 3 −1
, u = , v = , c = −2.
4.2 Matrix-vector multiplication and linear combina- tions
Recall the general definition of matrix-vector multiplication Ax is
a
11 a12 a13
21 a22 a23
2n
.
.
.
. . .
. . .
.
am1 am2 am3
· · ·
amn
xn
2 21 1 22 2
· · · a x a x + a x + · · · + a x
2n n
. .
. .
am1x1 + am2x2 + · · · + amnxn
a · · · a1n x1 a11x1 + a12x2 + · · · + a1nxn
=
(4.1)
34
x1v1 + x2v2 + · · · + xnvn =
1 21
x1am1
+
2 22
. .
. .
x2am2
+ · · · +
x a x a x a
n 2n
.
.
xnamn
The Matrix Equation Ax = b
There is an important way to decompose matrix-vector multiplication involving a linear combination. To see how, let v1, v2, . . . , vn denote the columns of A and consider the following linear combination:
x1a11 x2a12 xna1n
=
x a + x a + · · · + x a
1 11 2 12 n 1n
1 21 2 22
x a + x a + · · · + x a
n 2n
.
.
x1am1 + x2am2 + · · · + xnamn
.
(4.2)
1
· · ·
vn
and x = (x1, x2, . . . , xn) then
Ax = x1v1 + x2v2 + · · · + xnvn.
In summary, the vector Ax is a linear combination of the columns of A where the scalar in the linear combination are the components of x! This (important) observation gives an alternative way to compute Ax.
Example 4.5. Given
A =
−1 1
0
4 1 −2
3 −3 3
0 −2 −3
−1
−2
, x = 2 ,
compute Ax in two ways: (1) using the original Definition 4.1, and (2) as a linear combination of the columns of A.
4.3 The matrix equation problem
As we have seen, with a matrix A and any vector x, we can produce a new output vector via the multiplication Ax. If A is a m×n matrix then we must have x ∈ Rn and the output vector Ax is in Rm. We now introduce the following problem:
Problem: Given a matrix A ∈ Mm×n and a vector b ∈ Rm, find, if possible, a vector
x ∈ Rn such that
Ax = b. (⋆)
Equation (⋆) is a matrix equation where the unknown variable is x. If u is a vector such that Au = b, then we say that u is a solution to the equation Ax = b. For example,
35
Lecture 4
suppose that
A =
1 0
−3
1 0 7
, b = .
Does the equation Ax = b have a solution? Well, for any x =
x
1
x2
we have that
Ax =
1 0 x
1
=
x
1
1 0 x2 x1
and thus any output vector Ax has equal entries. Since b does not have equal entries then the equation Ax = b has no solution.
We now describe a systematic way to solve matrix equations. As we have seen, the vector Ax is a linear combination of the columns of A with the coefficients given by the components of x. Therefore, the matrix equation problem is equivalent to the linear combination problem. In Lecture 2, we showed that the linear combination problem can be solved by solving a system of linear equations. Putting all this together then, if A = v1 v2 · · · vn and b ∈ Rm then:
To find a vector x ∈ Rn that solves the matrix equation
Ax = b
we solve the linear system whose augmented matrix is
A
b = v1 v2 · · · vn b .
From now on, a system of linear equations such as
a11x1 a21x1 a31x1
+
+
+
a12x2 a22x2 a32x2
+
+
+
a13x3 a23x3 a33x3
a1nxn a2nxn a3nxn
= b1
= b2
= b3
.
.
+
.
.
.
.
+
+ · · · +
+ · · · +
+ · · · +
. .
+ · · · +
.
.
= bm
am1x1
am2x2 am3x3 amnxn
will be written in the compact form
Ax = b
where A is the coefficient matrix of the linear system, b is the output vector, and x is the unknown vector to be solved for. We summarize our findings with the following theorem.
Theorem 4.6: Let A ∈ Mm×n and b ∈ Rm. The following statements are equivalent:
36
The Matrix Equation Ax = b
Example 4.7. Solve, if possible, the matrix equation Ax = b if
−6 12
1 −2
3 −4
A = 1 5 2 , b = 4 .
−3 −7
Solution. First form the augmented matrix:
1
[A b] = 1
3 −4 −2
5 2 4
−3 −7 −6 12
Performing the row reduction algorithm we obtain that
1
1 5 2 4 ∼ 0
3 −4 −2 1 3 −4 −2
1 3 3 .
−3 −7 −6 12 0 0 −12 0
Here r = rank(A) = 3 and therefore d = 0, i.e., no free parameters. Peforming back substitution we obtain that x1 = −11, x2 = 3, and x3 = 0. Thus, the solution to the matrix equation is unique (no free parameters) and is given by
−11
x = 3
0
Let’s verify that Ax = b:
1
Ax = 1
5 2
3 −4 −11
−11 + 9 + 0
−3 −7 −6 0 33 − 21
−2
12
3 = −11 + 15 + 0 = 4 = b
In other words, b is a linear combination of the columns of A:
1 3 −4 −2
−3
7
−6
−11 1 + 3 5 + 0 2 = 4
12
37
Lecture 4
Example 4.8. Solve, if possible, the matrix equation Ax = b if
A =
1 2 3
2 4 −4
, b = .
1 2 3 −2R1 +R2 1 2 3
2 4 −4 0 0 −10
Solution. Row reducing the augmented matrix A b we get
−−−−−→ .
The last row is inconsistent and therefore there is no solution to the matrix equation Ax = b. In other words, b is not a linear combination of the columns of A.
Example 4.9. Solve, if possible, the matrix equation Ax = b if
A =
1 −1 2
2
0 3 6 −1
, b = .
Solution. First note that the unknown vector x is in R3 because A has n = 3 columns. The linear system Ax = b has m = 2 equations and n = 3 unknowns. The coefficient matrix A
has rank r = 2, and therefore the solution set will contain d = n − r = 1 parameter. The
augmented matrix A b is
A
b =
1 −1 2 2
0 3 6 −1
.
Let x3 = t be the parameter and use the last row to solve for x2:
2
1
3
x = − − 2t
Now use the first row to solve for x1:
1 2 3
1
3
x = 2 + x − 2x = 2 + (− − 2t) −
5
3
2t = − 4t.
Thus, the solution set to the linear system is
1
5
3
x = − 4t
2
1
3
x = − − 2t
x3 = t
where t is an arbitrary number. Therefore, the matrix equation Ax = b has an infinite number of solutions and they can all be written as
5
3
— 4t
1
3
x = − − 2t
t
38
x = −7/3
−1
The Matrix Equation Ax = b
where t is an arbitrary number. Equivalently, b can be written as a linear combination of the columns of A in infinitely many ways. For example, choosing t = −1 gives the particular solution
17/3
and you can verify that
17/3
A −7/3 = b.
−1
Recall from Definition 3.7 that the span of a set of vectors v1, v2, . . . , vp, which we denoted
by span{v1, v2, . . . , vp}, is the space of vectors that can be written as a linear combination
of the vectors v1, v2, . . . , vp.
Example 4.10. Is the vector b in the span of the vectors v1, v2?
b = 4 , v
0 3 −5
1 2
= −2 , v = 6
4 1 1
Solution. The vector b is in span{v1, v2} if we can find scalars x1, x2 such that
x1v1 + x2v2 = b.
If we let A ∈ R3×2 be the matrix
1 2
3
A = [v v ] = −
2 6
1 1
−5
then we need to solve the matrix equation Ax = b. Note that here x = Performing row reduction on the augmented matrix [A b] we get that
x
1
x2
2
∈ R .
−2 6 4
3 0 1
~ 0 1 1.5
−5 0 2.5
1 1 4 0 0 0
Therefore, the linear system is consistent and has solution
x =
2.5
1.5
Therefore, b is in span{v1, v2}, and b can be written in terms of v1 and v2 as
2.5v1 + 1.5v2 = b
39
Lecture 4
If v1, v2, . . . , vp are vectors in Rn and it happens to be true that span{v1, v2, . . . , vp} = Rn then we would say that the set of vectors {v1, v2, . . . , vp} spans all of Rn. From Theorem 4.6, we have the following.
m×n 1 2 n
Theorem 4.11: Let A ∈ M be a matrix with columns v , v , . . . , v , that is, A =
v1 v2 · · · vn . The following are equivalent:
Example 4.12. Do the vectors v1, v2, v3 span R3?
1 2
1
v = −
3 , v = −
2 3
4 , v = 2
5 2 3
−1
3
1 2 3 1
Solution. From Theorem 4.11, the vectors v , v , v span R if the matrix A = v v2
v3
has rank r = 3 (leading entries in its REF/RREF). The RREF of A is
2 −1 1 0
−3 −4 2 ∼ 0 1 0
1 0
5 2 3 0 0 1
which does indeed have r = 3 leading entries. Therefore, regardless of the choice of b ∈ R3, the augmented matrix [A b] will be consistent. Therefore, the vectors v1, v2, v3 span R3:
span{v1, v2, v3} = R3.
In other words, every vector b ∈ R3 can be written as a linear combination of v1, v2, v3.
After this lecture you should know the following:
combination of the columns of A, and when the augmented matrix A b is consistent
(Theorem 4.6)
40
The Matrix Equation Ax = b
Lecture 5
41
Lecture 5
Homogeneous and Nonhomogeneous Systems
5.1 Homogeneous linear systems
We begin with a definition.
Definition 5.1: A linear system of the form Ax = 0 is called a homogeneous linear system.
A homogeneous system Ax = 0 always has at least one solution, namely, the zero solution because A0 = 0. A homogeneous system is therefore always consistent. The zero solution x = 0 is called the trivial solution and any non-zero solution is called a nontrivial solution. From the existence and uniqueness theorem (Theorem 2.5), we know that a consistent linear system will have either one solution or infinitely many solutions. Therefore, a homogeneous linear system has nontrivial solutions if and only if its solution set has at least one parameter.
Recall that the number of parameters in the solution set is d = n − r, where r is the rank of the coefficient matrix A and n is the number of unknowns.
Example 5.2. Does the linear homogeneous system have any nontrivial solutions?
3x1 + x2 − 9x3 = 0 x1 + x2 − 5x3 = 0 2x1 + x2 − 7x3 = 0
Solution. The linear system will have a nontrivial solution if the solution set has at least one free parameter. Form the augmented matrix:
1 −9
1 1 −5 0
2 1 −7 0
3 0
42
Homogeneous and Nonhomogeneous Systems
The RREF is:
3
1 −9
0 1
0 −2
1 1 −5 0 ∼ 0 1 −3 0
0
2 1 −7 0 0 0 0 0
The system is consistent. The rank of the coefficient matrix is r = 2 and thus there will be
d = 3 − 2 = 1 free parameter in the solution set. If we let x3 be the free parameter, say
x3 = t, then from the row equivalent augmented matrix
0 −2
0 1 −3 0
1 0
0 0 0 0
we obtain that x2 = 3x3 = 3t and x1 = 2x3 = 2t. Therefore, the general solution of the linear system is
x1 = 2t x2 = 3t x3 = t
The general solution can be written in vector notation as
2
x = 3 t
1
2
1
Or more compactly if we let v = 3 then x = vt. Hence, any solution x to the linear
2
system can be written as a linear combination of the vector v = 3 . In other words, the
1
solution set of the linear system is the span of the vector v:
span{v}.
Notice that in the previous example, when solving a homogeneous system Ax = 0 using
row reduction, the last column of the augmented matrix A 0 remains unchanged (always
0) after every elementary row operation. Hence, to solve a homogeneous system, we can row reduce the coefficient matrix A only and then set all rows equal to zero when performing back substitution.
Example 5.3. Find the general solution of the homogenous system Ax = 0 where
1 4
2 2 1
A = 3 7 7 3 13 .
2 5 5 2 9
43
Lecture 5
Solution. After row reducing we obtain
1
4 1
2 2 1
A = 3 7 7 3 13 ∼ 0
2 5 5
0 0 1
1 1 0 1
2 9 0 0 0 0 0
2
x =
−t − t
2 1
t2 t3 t1
1
0
1
2
0
0
v v
1 2
−1 −1 0
1
0
Here n = 5, and r = 2, and therefore the number of parameters in the solution set is
d = n − r = 3. The second row of rref(A) gives the equation
x2 + x3 + x5 = 0.
Setting x5 = t1 and x3 = t2 as free parameters we obtain that
x2 = −x3 − x5 = −t2 − t1.
From the first row we obtain the equation
x1 + x4 + 2x5 = 0
The unknown x5 has already been assigned, so we must now choose either x1 or x4 to be a parameter. Choosing x4 = t3 we obtain that
x1 = −x4 − 2x5 = −t3 − 2t1
In summary, the general solution can be written as
−t3 − 2t1 −2 0 −1
` ˛¸ x ` ˛¸ x ` ˛¸ x
v
3
= t 0 +t 1 +t 0 = t v + t v + t v
3 1 1 2 2 3 3
where t1, t2, t3 are arbitrary parameters. In other words, any solution x is in the span of
v1, v2, v3:
x ∈ span{v1, v2, v3}.
The form of the general solution in Example 5.3 holds in general and is summarized in the following theorem.
Theorem 5.4: Consider the homogenous linear system Ax = 0, where A ∈ Mm×n and
0 ∈ Rm. Let r be the rank of A.
x = t1v1 + t2v2 + · · · + tpvd.
44
Homogeneous and Nonhomogeneous Systems
In other words, any solution x is in the span of v1, v2, . . . , vd:
x ∈ span{v1, v2, . . . , vd}.
A solution x to a homogeneous system written in the form
x = t1v1 + t2v2 + · · · + tpvd
is said to be in parametric vector form.
5.2 Nonhomogeneous systems
As we have seen, a homogeneous system Ax = 0 is always consistent. However, if b is non- zero, then the nonhomogeneous linear system Ax = b may or may not have a solution. A natural question arises: What is the relationship between the solution set of the homogeneous system Ax = 0 and that of the nonhomogeneous system Ax = b when it is consistent? To answer this question, suppose that p is a solution to the nonhomogeneous system Ax = b, that is, Ap = b. And suppose that v is a solution to the homogeneous system Ax = 0, that is, Av = 0. Now let q = p + v. Then
Aq = A(p + v)
= Ap + Av
= b + 0
= b.
Therefore, Aq = b. In other words, q = p + v is also a solution of Ax = b. We have therefore proved the following theorem.
Theorem 5.5: Suppose that the linear system Ax = b is consistent and let p be a solution. Then any other solution q of the system Ax = b can be written in the form q = p + v, for some vector v that is a solution to the homogeneous system Ax = 0.
Another way of stating Theorem 5.5 is the following: If the linear system Ax = b is consistent and has solutions p and q, then the vector v = q−p is a solution to the homogeneous system Ax = 0. The proof is a simple computation:
Av = A(q − p) = Aq − Ap = b − b = 0.
More generally, any solution of Ax = b can be written in the form
q = p + t1v1 + t2v2 + · · · + tpvd
where p is one particular solution of Ax = b and the vectors v1, v2, . . . , vd span the solution set of the homogeneous system Ax = 0.
45
b 0
p
b
b b tv
b v
span{v}
Lecture 5
There is a useful geometric interpretation of the solution set of a general linear system. We saw in Lecture 3 that we can interpret the span of a set of vectors as a plane containing the zero vector 0. Now, the general solution of Ax = b can be written as
x = p + t1v1 + t2v2 + · · · + tpvd.
Therefore, the solution set of Ax = b is a shift of the span{v1, v2, . . . , vd} by the vector p. This is illustrated in Figure 5.1.
p + span{v}
p + tv
Figure 5.1: The solution sets of a homogeneous and nonhomogeneous system.
Example 5.6. Write the general solution, in parametric vector form, of the linear system 3x1 + x2 − 9x3 = 2
x1 + x2 − 5x3 = 0 2x1 + x2 − 7x3 = 1.
Solution. The RREF of the augmented matrix is:
1 −9 0 −2
1 1 −5 0 ∼ 0
2 1 −7 1 0
1 −3 −1
0 0 0
3 2 1 1
t
x = 3t − 1 = −
1 +t 3
The system is consistent and the rank of the coefficient matrix is r = 2. Therefore, there are d = 3 − 2 = 1 parameters in the solution set. Letting x3 = t be the parameter, from the second row of the RREF we have
x2 = 3t − 1
And from the first row of the RREF we have
x1 = 2t + 1
Therefore, the general solution of the system in parametric vector form is
2t + 1 1 2
0 1
` ˛p¸ x `˛v¸x
46
Homogeneous and Nonhomogeneous Systems
You should check that p = (1, −1, 0) solves the linear system Ax = b, and that v = (2, 3, 1) solves the homogeneous system Ax = 0.
Example 5.7. Write the general solution, in parametric vector form, of the linear system represented by the augmented matrix
−3 6
3 3
−1 1 −2 −1 .
−1 1 −2 −1
2 −2 4 2
Solution. Write the general solution, in parametric vector form, of the linear system repre- sented by the augmented matrix
3 −3 6 3
2 −2 4 2
The RREF of the augmented matrix is
3
3 1
−1 2
−3 6
−1 1 −2 −1 ∼ 0
2 −2 4
0 0 0
2 0 0 0 0
1
Here n = 3, r = 1 and therefore the solution set will have d = 2 parameters. Let x3 = t1
and x2 = t2. Then from the first row we obtain
x1 = 1 + x2 − 2x3 = 1 + t2 − 2t1
The general solution in parametric vector form is therefore
1 −2 1
1 2
x = 0 +t 0 +t 1
0 1 0
`˛p¸x ` ˛v¸1 x `˛v¸2 x
You should verify that p is a solution to the linear system Ax = b:
Ap = b
And that v1 and v2 are solutions to the homogeneous linear system Ax = 0:
Av1 = Av2 = 0
47
Lecture 5
The material in this lecture is so important that we will summarize the main results. The solution set of a linear system Ax = b can be written in the form
x = p + t1v1 + t2v2 + · · · + tdvd
where Ap = b and where each of the vectors v1, v2, . . . , vd satisfies Avi = 0. Loosely speaking,
{Solution set of Ax = b} = p + {Solution set of Ax = 0}
or
{Solution set of Ax = b} = p + span{v1, v2, . . . , vd}
where p satisfies Ap = b and Avi = 0.
After this lecture you should know the following:
and the homogeneous equation Ax = 0
48
Homogeneous and Nonhomogeneous Systems
Lecture 6
49
Lecture 6
Linear Independence
6.1 Linear independence
In Lecture 3, we defined the span of a set of vectors {v1, v2, . . . , vn} as the collection of all possible linear combinations
t1v1 + t2v2 + · · · + tnvn
and we denoted this set as span{v1, v2, . . . , vn}. Thus, if x ∈ span{v1, v2, . . . , vn} then by definition there exists scalars t1, t2, . . . , tn such that
x = t1v1 + t2v2 + · · · + tnvn.
A natural question that arises is whether or not there are multiple ways to express x as a linear combination of the vectors v1, v2, . . . , vn. For example, if v1 = (1, 2), v2 = (0, 1), v3 = (−1, −1), and x = (3, −1) then you can verify that x ∈ span{v1, v2, v3} and x can be written in infinitely many ways using v1, v2, v3. Here are three ways:
x = 3v1 − 7v2 + 0v3
x = −4v1 + 0v2 − 7v3
x = 0v1 − 4v2 − 3v3.
The fact that x can be written in more than one way in terms of v1, v2, v3 suggests that there might be a redundancy in the set {v1, v2, v3}. In fact, it is not hard to see that v3 = −v1 +v2, and thus v3 ∈ span{v1, v2}. The preceding discussion motivates the following definition.
Definition 6.1: A set of vectors {v1, v2, . . . , vn} is said to be linearly dependent if some vj can be written as a linear combination of the other vectors, that is, if
vj ∈ span{v1, . . . , vj−1, vj+1, . . . , vn}.
If {v1, v2, . . . , vn} is not linearly dependent then we say that {v1, v2, . . . , vn} is linearly independent.
50
Linear Independence
Example 6.2. Consider the vectors
1
3
v = 2 , v
1 2
4
6
= 5 , v
3
2
0
= 1 .
Show that they are linearly dependent.
Solution. By inspection, we have
2 2 4
2v + v = 4 + 1 = 5 = v
1 3 2
6 0 6
Thus, v2 ∈ span{v1, v3} and therefore {v1, v2, v3} is linearly dependent.
Notice that in the previous example, the equation 2v1 + v3 = v2 is equivalent to
2v1 − v2 + v3 = 0.
Hence, because {v1, v2 v3} is a linearly dependent set, it is possible to write the zero vector 0 as a linear combination of {v1, v2 v3} where not all the coefficients in the linear combination are zero. This leads to the following characterization of linear independence.
Theorem 6.3: The set of vectors {v1, v2, . . . , vn} is linearly independent if and only if 0
can be written in only one way as a linear combination of {v1, v2, . . . , vn}. In other words,
if
t1v1 + t2v2 + · · · + tnvn = 0
then necessarily the coefficients t1, t2, . . . , tn are all zero.
Proof. If {v1, v2, . . . , vn} is linearly independent then every vector x ∈ span{v1, v2, . . . , vn} can be written uniquely as a linear combination of {v1, v2, . . . , vn}, and this applies to the particular case of the zero vector x = 0.
Now assume that 0 can be written uniquely as a linear combination of {v1, v2, . . . , vn}.
In other words, assume that if
t1v1 + t2v2 + · · · + tnvn = 0
then t1 = t2 = · · · = tn = 0. Now take any x ∈ span{v1, v2, . . . , vn} and suppose that there are two ways to write x in terms of {v1, v2, . . . , vn}:
r1v1 + r2v2 + · · · + rnvn = x
s1v1 + s2v2 + · · · + snvn = x.
Subtracting the second equation from the first we obtain that
(r1 − s1)v1 + (r2 − s2)v2 + · · · + (rn − sn)vn = x − x = 0.
51
Lecture 6
The above equation is a linear combination of v1, v2, . . . , vn resulting in the zero vector 0. But we are assuming that the only way to write 0 in terms of {v1, v2, . . . , vn} is if all the
coefficients are zero. Therefore, we must have r1 − s1 = 0, r2 − s2 = 0, . . . , rn − sn = 0, or
equivalently that r1 = s1, r2 = s2, . . . , rn = sn. Therefore, the linear combinations
r1v1 + r2v2 + · · · + rnvn = x
s1v1 + s2v2 + · · · + snvn = x
are actually the same. Therefore, each x ∈ span{v1, v2, . . . , vn} can be written uniquely in terms of {v1, v2, . . . , vn}, and thus {v1, v2, . . . , vn} is a linearly independent set.
Because of Theorem 6.3, an alternative definition of linear independence of a set of vectors
{v1, v2, . . . , vn} is that the vector equation
x1v1 + x2v2 + · · · + xnvn = 0
has only the trivial solution, i.e., the solution x1 = x2 = · · · = xn = 0. Thus, if {v1, v2, . . . , vn}
is linearly dependent, then there exist scalars x1, x2, . . . , xn not all zero such that
x1v1 + x2v2 + · · · + xnvn = 0.
Hence, if we suppose for instance that xn /= 0 then we can write vn in terms of the vectors
v1, . . . , vn−1 as follows:
n
x
1
x
2
xn xn
1 2
x
n−1
xn
v = − v − v − · · · − v
n−1
.
In other words, vn ∈ span{v1, v2, . . . , vn−1}.
According to Theorem 6.3, the set of vectors {v1, v2, . . . , vn} is linearly independent if
the equation
x1v1 + x2v2 + · · · + xnvn = 0 (6.1)
has only the trivial solution. Now, the vector equation (6.1) is a homogeneous linear system of equations with coefficient matrix
A = v1 v2 · · · vn .
Therefore, the set {v1, v2, . . . , vn} is linearly independent if and only if the the homogeneous system Ax = 0 has only the trivial solution. But the homogeneous system Ax = 0 has only the trivial solution if there are no free parameters in its solution set. We therefore have the following.
Theorem 6.4: The set {v1, v2, . . . , vn} is linearly independent if and only if the the rank of A is r = n, that is, if the number of leading entries r in the REF (or RREF) of A is exactly n.
Example 6.5. Are the vectors below linearly independent?
0 1 4
1
v =
5
8
2 3
1 , v = 2 , v =
−1
0
52
Linear Independence
Solution. Let A be the matrix
0 1
A = v1 v2 v3 = 1 2 −1
5 8 0
4
Performing elementary row operations we obtain
1
A ∼ 0
1 4
0 0 13
2 −1
Clearly, r = rank(A) = 3, which is equal to the number of vectors n = 3.
{v1, v2, v3} is linearly independent.
Example 6.6. Are the vectors below linearly independent?
1 4 2
Therefore,
1
v =
3
2
2 , v 5
6
3
= , v = 1
0
Solution. Let A be the matrix
1 4
A = v1 v2 v3 = 2 5 1
2
3 6 0
Performing elementary row operations we obtain
1
4
A ∼ 0
2
−3 −3
0 0 0
Clearly, r = rank(A) = 2, which is not equal to the number of vectors, n = 3. Therefore,
{v1, v2, v3} is linearly dependent. We will find a nontrivial linear combination of the vectors
v1, v2, v3 that gives the zero vector 0. The REF of A = [v1 v2 v3] is
4
A ∼ 0 −3 −3
1 2
0 0 0
Since r = 2, the solution set of the linear system Ax = 0 has d = n − r = 1 free parameter.
Using back substitution on the REF above, we find that the general solution of Ax = 0
written in parametric form is
x = t −
2
1
1
The vector
v =
2
−1
1
53
Lecture 6
spans the solution set of the system Ax = 0. Choosing for instance t = 2 we obtain the solution
x = t −
2 4
1
1 = −
2
2 .
Therefore,
4v1 − 2v2 + 2v3 = 0
is a non-trivial linear combination of v1, v2, v3 that gives the zero vector 0. And, for instance,
v3 = −2v1 + v2
that is, v3 ∈ span{v1, v2}.
Below we record some simple observations on the linear independence of simple sets:
is non-zero then
tv1 = 0
is true if and only if t = 0.
tv1 − v2 = 0
is a non-trivial linear combination of v1, v2 giving the zero vector 0.
0v1 + 0v2 + · · · + 0vp−1 + 2vp = 0
is a non-trivial linear combination giving the zero vector 0.
6.2 The maximum size of a linearly independent set
The next theorem puts a constraint on the maximum size of a linearly independent set in
Rn.
Theorem 6.7: Let {v1, v2, . . . , vp} be a set of vectors in Rn. If p > n then v1, v2, . . . , vp are linearly dependent. Equivalently, if the vectors v1, v2, . . . , vp in Rn are linearly inde- pendent then p ≤ n.
54
Linear Independence
1
Proof. Let A = v v2
p
· · · v . Thus, A is a n × p matrix. Since A has n rows, the
maximum rank of A is n, that is r ≤ n. Therefore, the number of free parameters d = p − r is always positive because p > n ≥ r. Thus, the homogeneous system Ax = 0 has non-trivial solutions. In other words, there is some non-zero vector x ∈ Rp such that
Ax = x1v1 + x2v2 + · · · + xpvp = 0
and therefore {v1, v2, . . . , vp} is linearly dependent.
Theorem 6.7 will be used when we discuss the notion of the dimension of a space. Although we have not discussed the meaning of dimension, the above theorem says that in n-dimensional space Rn, a set of vectors {v1, v2, . . . , vp} consisting of more than n vectors is automatically linearly dependent.
Example 6.8. Are the vectors below linearly independent?
8 4 2 3
1
v =
0
2
, v =
3 11
−4
, v3 =
0
1
, v4 =
−9
−5
−2 6 1 3
5
, v =
0
−2
−7
7
.
1 2 3 4 5
4
1 5
Solution. The vectors v , v , v , v , v are in R . Therefore, by Theorem 6.7, the set {v , . . . , v }
is linearly dependent. To see this explicitly, let A = v1 v2 v3 v4 v5 . Then
A ∼
1
0 0 0 −1
0 1 0 0 1
0 0 1 0 0
0 0 0 1 −2
One solution to the linear system Ax = 0 is x = (−1, 1, 0, −2, −1) and therefore
(−1)v1 + (1)v2 + (0)v3 + (−2)v4 + (−1)v5 = 0
Example 6.9. Suppose that the set {v1, v2, v3, v4} is linearly independent. Show that the set {v1, v2, v3} is also linearly independent.
Solution. We must argue that if there exists scalars x1, x2, x3 such that
x1v1 + x2v2 + x3v3 = 0
then necessarily x1, x2, x3 are all zero. Suppose then that there exists scalars x1, x2, x3 such that
x1v1 + x2v2 + x3v3 = 0.
Then clearly it holds that
x1v1 + x2v2 + x3v3 + 0v4 = 0.
But the set {v1, v2, v3, v4} is linearly independent, and therefore, it is necessary that x1, x2, x3
are all zero. This proves that v1, v2, v3 are also linearly independent.
55
Lecture 6
The previous example can be generalized as follows: If {v1, v2, . . . , vd} is linearly inde- pendent then any (non-empty) subset of the set {v1, v2, . . . , vd} is also linearly independent.
After this lecture you should know the following:
1 2 p
set of the homogeneous system Ax = 0, where A = v1 v2 · · · vp
56
Linear Independence
Lecture 7
57
Lecture 7
Introduction to Linear Mappings
7.1 Vector mappings
By a vector mapping we mean simply a function
T : Rn → Rm.
The domain of T is Rn and the co-domain of T is Rm. The case n = m is allowed of course. In engineering or physics, the domain is sometimes called the input space and the co-domain is called the output space. Using this terminology, the points x in the domain are called the inputs and the points T(x) produced by the mapping are called the outputs.
Definition 7.1: The vector b ∈ Rm is in the range of T, or in the image of T, if there exists some x ∈ Rn such that T(x) = b.
In other words, b is in the range of T if there is an input x in the domain of T that outputs b = T(x). In general, not every point in the co-domain of T is in the range of T. For example, consider the vector mapping T : R2 → R2 defined as
T(x) =
2
1
2
x sin(x )
2
1
— cos(x − 1)
" #
1 2
x2 + x2 + 1
.
The vector b = (3, −1) is not in the range of T because the second component of T(x) is positive. On the other hand, b = (−1, 2) is in the range of T because
T
2 2
1 1 sin(0) − cos(1 − 1)
−1
0 12 + 02 + 1 2
= = = b.
Hence, a corresponding input for this particular b is x = (1, 0). In Figure 7.1 we illustrate the general setup of how the domain, co-domain, and range of a mapping are related. A crucial idea is that the range of T may not equal the co-domain.
58
Introduction to Linear Mappings
x
b T(x)
Range
Rm, Co-domain
Rn, domain
Figure 7.1: The domain, co-domain, and range of a mapping.
For our purposes, vector mappings T : Rn → Rm can be organized into two categories: (1) linear mappings and (2) nonlinear mappings.
Definition 7.2: The vector mapping T : Rn → Rm is said to be linear if the following conditions hold:
As an example, the mapping
T(x) =
"
2
1
2
2
1
x sin(x ) − cos(x − 1)
1 2
x2 + x2 + 1
#
is nonlinear. To see this, previously we computed that
T
0
=
1 −1
2
.
59
T
0
= T 3
3 1
0
= 3T
1
0
= 3
−1
2
=
−3
6
.
However,
T
3
0
=
2 2
3 sin(0) − cos(3 −
1)
32 + 02 + 1
=
— cos(8)
10
/=
−3
6
.
Example 7.3. Is the vector mapping T : R2 → R3 linear?
T
x
1
x2
2x − x
1 2
= x + x
1 2
−x1 − 3x2
Solution. We must verify that the two conditions in Definition 7.2 hold. For the first condi- tion, take arbitrary vectors u = (u1, u2) and v = (v1, v2). We compute:
T (u + v) = T
u + v
1 1
u2 + v2
=
1 1 2 2
2(u + v ) − (u + v )
(u1 + v1) + (u2 + v2)
−(u1 + v1) − 3(u2 + v2)
=
2u + 2v − u − v
1 1 2 2
u1 + v1 + u2 + v2
−u1 − v1 − 3u2 − 3v2
=
2u − u + 2v − v
1 2 1 2
u1 + u2 + v1 + v2
−u1 − 3u2 − v1 − 3v2
2u − u
−u1 − 3u2
2v − v
1 2 1 2
= u1 + u2 + v1 + v2
−v1 − 3v2
= T(u) + T(v)
60
Introduction to Linear Mappings
Therefore, for arbitrary u, v ∈ R2, it holds that
T(u + v) = T(u) + T(v).
To prove the second condition, let c ∈ R be an arbitrary scalar. Then:
T(cu) = T
cu
1
cu2
1
2
2(cu ) − (cu )
1 2
= (cu ) + (cu )
−(cu1) − 3(cu2)
=
1 2
c(2u − u )
c(u1 + u2)
c(−u1 − 3u2)
2u − u
1 2
= c u1 + u2
−u1 − 3u2
= cT(u)
Therefore, both conditions of Definition 7.2 hold, and thus T is a linear map.
Example 7.4. Let α ≥ 0 and define the mapping T : Rn → Rn by the formula T(x) = αx. If 0 ≤ α ≤ 1 then T is called a contraction and if α > 1 then T is called a dilation. In either case, show that T is a linear mapping.
Solution. Let u and v be arbitrary. Then
T(u + v) = α(u + v) = αu + αv = T(u) + T(v).
This shows that condition (1) in Definition 7.2 holds. To show that the second condition holds, let c is any number. Then
T(cx) = α(cx) = αcx = c(αx) = cT(x).
Therefore, both conditions of Definition 7.2 hold, and thus T is a linear mapping. To see a
2
particular example, consider the case α = 1 and n = 3. Then,
1
2
1
2
x
1
1
2
2
1
2
x
3
T(x) = x = x .
61
Lecture 7
Given a matrix A ∈ Rm×n and a vector x ∈ Rn, in Lecture 4 we defined matrix-vector multiplication between A and x as an operation that produces a new output vector Ax ∈ Rm. We discussed that we could interpret A as a mapping that takes the input vector x ∈ Rn and produces the output vector Ax ∈ Rm. We can therefore associate to each matrix A a vector mapping T : Rn → Rm defined by
T(x) = Ax.
Such a mapping T will be called a matrix mapping corresponding to A and when con- venient we will use the notation TA to indicate that TA is associated to A. We proved in Lecture 4 (Theorem 4.3), that for any u, v ∈ Rn, and scalar c, matrix-vector multiplication satisfies the properties:
The following theorem is therefore immediate.
Theorem 7.5: To a given matrix A ∈ Rm×n associate the mapping T : Rn → Rm defined by the formula T(x) = Ax. Then T is a linear mapping.
Example 7.6. Is the vector mapping T : R2 → R3 linear?
T
x
1
x2
2x − x
1 2
= x + x
1 2
−x1 − 3x2
Solution. In Example 7.3 we showed that T is a linear mapping using Definition 7.2. Alter- natively, we observe that T is a mapping defined using matrix-vector multiplication because
T
x
1
x2
1 2
−x − 3x
1 2
2x − x 2 −1
= x1 + x2 = 1
1
−1 −3
x
1
x2
Therefore, T is a matrix mapping corresponding to the matrix
A =
2
1 1
−1
−1 −3
that is, T(x) = Ax. By Theorem 7.5, T is a linear mapping.
Introduction to Linear Mappings
Let T : Rn → Rm be a vector mapping. Recall that b ∈ Rm is in the range of T if there is some input vector x ∈ Rn such that T(x) = b. In this case, we say that b is the image of x under T or that x is mapped to b under T. If T is a nonlinear mapping, finding a specific vector x such that T(x) = b is generally a difficult problem. However, if T(x) = Ax is a matrix mapping, then it is clear that finding such a vector x is equivalent to solving the matrix equation Ax = b. In summary, we have the following theorem.
Theorem 7.7: Let T : Rn → Rm be a matrix mapping corresponding to A, that is, T(x) = Ax. Then b ∈ Rm is in the range of T if and only if the matrix equation Ax = b has a solution.
Let TA : Rn → Rm be a matrix mapping, that is, TA(x) = Ax. We proved that the output vector Ax is a linear combination of the columns of A where the coefficients in the linear combination are the components of x. Explicitly, if A = [v1 v2 · · · vn] and the
components of x = (x1, x2, . . . , xn) then
Ax = x1v1 + x2v2 + · · · + xnvn.
Therefore, the range of the matrix mapping TA(x) = Ax is
Range(TA) = span{v1, v2, . . . , vn}.
In words, the range of a matrix mapping is the span of its columns. Therefore, if v1, v2, . . . , vn
span all of Rm then every vector b ∈ Rm is in the range of TA.
Example 7.8. Let
A = 1
3 −4
1 −2
5 2 , b = 4 .
−3 −7 −6 12
Is the vector b in the range of the matrix mapping T(x) = Ax?
Solution. From Theorem 7.7, b is in the range of T if and only if the the matrix equation
Ax = b has a solution. To solve the system Ax = b, row reduce the augmented matrix
A b :
1 3 −4 −2 1 3 −4 −2
1 5 2 4 ∼ 0 1 3 3
−3 −7 −6 12 0 0 −12 0
The system is consistent and the (unique) solution is x = (−11, 3, 0). Therefore, b is in the
range of T.
7.4 Examples
If T : Rn
62
→ Rm is a linear mapping, then for any vectors v1, v2, . . . , vp and scalars
c1, c2, . . . , cp, it holds that
T(c1v1 + c2v2 + · · · + cpvd) = c1T(v1) + c2T(v2) + · · · + cdT(vp).
(⋆)
63
Lecture 7
Therefore, if all you know are the values T(v1), T(v2), . . . , T(vp) and T is linear, then you can compute T(v) for every
v ∈ span{v1, v2, . . . , vp}.
Example 7.9. Let T : R2 → R2 be a linear transformation that maps u to T(u) = (3, 4)
and maps v to T(v) = (−2, 5). Find T(2u + 3v).
Solution. Because T is a linear mapping we have that
T(2u + 3v) = T(2u) + T(3v) = 2T(u) + 3T(v).
We know that T(u) = (3, 4) and T(v) = (−2, 5). Therefore,
4
T(2u + 3v) = 2T(u) + 3T(v) = 2 + 3
5
=
3 −2 0
23
.
α
θ
b v
Example 7.10. (Rotations) Let Tθ : R2 → R2 be the mapping on the 2D plane that rotates every v ∈ R2 by an angle θ. Write down a formula for Tθ and show that Tθ is a linear mapping.
Tθ (v)
Solution. If v = (cos(α), sin(α)) then
Tθ(v) =
"
cos(α + θ)
sin(α + θ)
Then from the angle sum trigonometric identities:
#
.
" # "
cos(α + θ) cos(α) cos(θ) − sin(α) sin(θ)
Tθ (v) = =
sin(α + θ) cos(α) sin(θ) + sin(α) cos(θ)
#
But
Tθ (v) =
"
cos(α) cos(θ) − sin(α) sin(θ)
cos(α) sin(θ) + sin(α) cos(θ)
# " # "
=
cos(θ) − sin(θ) cos(α)
sin(θ)
cos(θ) sin(α)
#
` ˛v¸ x
.
64
Introduction to Linear Mappings
If we scale v by any c > 0 then performing the same computation as above we obtain that
Tθ (cv) = cT(v). Therefore, Tθ is a matrix mapping with corresponding matrix
A =
cos(θ)
— sin(θ)
cos(θ)
" #
sin(θ)
.
Thus, Tθ is a linear mapping.
Example 7.11. (Projections) Let T : R3 → R2 be the vector mapping
1
T x
x x
1
2 2
= x .
x3 0
Show that T is a linear mapping and describe the range of T.
Solution. First notice that
1
T x
1
2 2
= x = 0
1
2
x x 1 0 0 x
1
x3 0 0 0 0 x3
0 x .
Thus, T is a matrix mapping corresponding to the matrix
0
0 0 0
1 0
A = 0 1 0 .
Therefore, T is a linear mapping. Geometrically, T takes the vector x and projects it to the (x1, x2) plane, see Figure 7.2. What is the range of T? The range of T consists of all vectors in R3 of the form
b = s
t
0
where the numbers t and s are arbitrary. For each b in the range of T, there are infinitely many x’s such that T(x) = b.
b
b x = x
2
x3
x1
x
1
T(x) = x
b
2
0
Figure 7.2: Projection onto the (x1, x2) plane
65
Lecture 7
After this lecture you should know the following:
66
Introduction to Linear Mappings
Lecture 8
67
Lecture 8
Onto and One-to-One Mappings, and the Matrix of a Linear Mapping
8.1 Onto Mappings
We have seen through examples that the range of a vector mapping (linear or nonlinear) is not always the entire co-domain. For example, if TA(x) = Ax is a matrix mapping and b is such that the equation Ax = b has no solutions then the range of T does not contain b and thus the range is not the whole co-domain.
Definition 8.1: A vector mapping T : Rn → Rm is said to be onto if for each b ∈ Rm
there is at least one x ∈ Rn such that T(x) = b.
For a matrix mapping TA(x) = Ax, the range of TA is the span of the columns of A. Therefore:
Theorem 8.2: Let TA : Rn → Rm be the matrix mapping TA(x) = Ax, where A ∈
Mm×n. Then TA is onto if and only if the columns of A span all of Rm.
Combining Theorem 4.11 and Theorem 8.2 we have:
Theorem 8.3: Let TA : Rn → Rm be the matrix mapping TA(x) = Ax, where A ∈ Rm×n. Then TA is onto if and only if r = rank(A) = m.
Example 8.4. Let T : R3 → R3 be the matrix mapping with corresponding matrix
1
A = −3 −4 2
5 2 3
2 −1
Is TA onto?
68
Onto, One-to-One, and Standard Matrix
Solution. The rref(A) is
1
2 −1 1 0
−3 −4 2 ∼ 0 1 0
5 2 3 0 0 1
0
Therefore, r = rank(A) = 3. The dimension of the co-domain is m = 3 and therefore TA is onto. Therefore, the columns of A span all of R3, that is, every b ∈ R3 can be written as a linear combination of the columns of A:
1
span −
2
2
3 , −4 , 2
3
2 −1
=
R3
Example 8.5. Let TA : R4 → R3 be the matrix mapping with corresponding matrix
2 −1
A = −1 4 1 8
2 0 −2 0
1 4
Is TA onto?
Solution. The rref(A) is
1
2 −1
4 1
0 −1
A = −1 4 1 8 ∼ 0 1 0 2
2 0 −2 0 0 0 0 0
0
Therefore, r = rank(A) = 2. The dimension of the co-domain is m = 3 and therefore TA is not onto. Notice that v3 = −v1 and v4 = 2v2. Thus, v3 and v4 are already in the span of the columns v1, v2. Therefore,
span{v1, v2, v3, v4} = span{v1, v2} =
/ R3.
Below is a theorem which places restrictions on the size of the domain of an onto mapping.
Theorem 8.6: Suppose that TA : Rn → Rm is a matrix mapping corresponding to
A ∈ Mm×n. If TA is onto then m ≤ n.
Proof. If TA is onto then the rref(A) has r = m leading 1’s. Therefore, A has at least m
columns. The number of columns of A is n. Therefore, m ≤ n.
An equivalent way of stating Theorem 8.6 is the following.
69
Lecture 8
Corollary 8.7: If TA : Rn → Rm is a matrix mapping corresponding to A ∈ Mm×n and
n < m then TA cannot be onto.
Intuitively, if the domain Rn is “smaller” than the co-domain Rm and TA : Rn → Rm is linear then TA cannot be onto. For example, a matrix mapping TA : R → R2 cannot be onto. Linearity plays a key role in this. In fact, there exists a continuous (nonlinear) function f : R → R2 whose range is a square! In this case, the domain is 1-dimensional and the range is 2-dimensional. This situation cannot happen when the mapping is linear.
Example 8.8. Let TA : R2 → R3 be the matrix mapping with corresponding matrix
1 4
A = −3 2
2 1
Is TA onto?
Solution. TA is onto because the domain is R2 and the co-domain is R3. Intuitively, two vectors are not enough to span R3. Geometrically, two vectors in R3 span a 2D plane going through the origin. The vectors not on the plane span{v1, v2} are not in the range of TA.
8.2 One-to-One Mappings
Given a linear mapping T : Rn → Rm, the question of whether b ∈ Rm is in the range of T is an existence question. Indeed, if b ∈ Range(T) then there exists a x ∈ Rm such that T(x) = b. We now want to look at the problem of whether x is unique. That is, does there exist a distinct y such that T(y) = b.
Definition 8.9: A vector mapping T : Rn → Rm is said to be one-to-one if for each
b ∈ Range(T) there exists only one x ∈ Rn such that T(x) = b.
When T is a linear mapping, we have all the tools necessary to give a complete description of when T is one-to-one. To do this, we use the fact that if T : Rn → Rm is linear then T(0) = 0. Here is one proof: T(0) = T(x − x) = T(x) − T(x) = 0.
Theorem 8.10: Let T : Rn → Rm be linear. Then T is one-to-one if and only if T(x) = 0
implies that x = 0.
If TA : Rn → Rm is a matrix mapping then according to Theorem 8.10, TA is one-to-one if and only if the only solution to Ax = 0 is x = 0. We gather these facts in the following theorem.
70
Onto, One-to-One, and Standard Matrix
Theorem 8.11: Let TA : Rn → Rm be a matrix mapping, where A = [v1 v2 · · · vn] ∈
Mm×n. The following statements are equivalent:
Example 8.12. Let TA : R4 → R3 be the matrix mapping with matrix
−2 6
2 −2 0 2
3 4
A = −1 0 −2 −1 .
Is TA one-to-one?
Solution. By Theorem 8.11, TA is one-to-one if and only if the columns of A are linearly independent. The columns of A lie in R3 and there are n = 4 columns. From Lecture 6, we know then that the columns are not linearly independent. Therefore, TA is not one-to-one. Alternatively, A will have rank at most r = 3 (why?). Therefore, the solution set to Ax = 0 will have at least one parameter, and thus there exists infinitely many solutions to Ax = 0. Intuitively, because R4 is “larger” than R3, the linear mapping TA will have to project R4 onto R3 and thus infinitely many vectors in R4 will be mapped to the same vector in R3.
Example 8.13. Let TA : R2 → R3 be the matrix mapping with matrix
1 0
A = 3 −1
2 0
Is TA one-to-one?
Solution. By inspection, we see that the columns of A are linearly independent. Therefore,
TA is one-to-one. Alternatively, one can compute that
1 0
rref(A) = 0 1
0 0
Therefore, r = rank(A) = 2, which is equal to the number columns of A.
Lecture 8
8.3 Standard Matrix of a Linear Mapping
We have shown that all matrix mappings TA are linear mappings. We now want to answer the reverse question: Are all linear mappings matrix mappings in disguise? If T : Rn → Rm is a linear mapping, then to show that T is in fact a matrix mapping we must show that there is some matrix A ∈ Mm×n such that T(x) = Ax. To that end, introduce the standard unit vectors e1, e2, . . . , en in Rn:
1 0 0 0
1
e =
0
.
.
0
2
, e =
0
.
.
1
3
, e =
1
.
.
0
0
n
, · · · , e =
0 0
Every x ∈ Rn is in span{e1, e2, . . . , en} because:
x1 1 0 0
0
.
.
1
0
.
2
x =
= x
1
x 0
+ x
2
1
+ · · · + x
n
0
. . . .
. . . .
1
= x1e1 + x2e2 + · · · + xnen
xn 0 0
With this notation we prove the following.
Theorem 8.14: Every linear mapping is a matrix mapping.
Proof. Let T : Rn → Rm be a linear mapping. Let
v1 = T(e1), v2 = T(e2), . . . , vn = T(en).
The co-domain of T is Rm, and thus vi ∈ Rm. Now, for arbitrary x ∈ Rn we can write
x = x1e1 + x2e2 + · · · + xnen.
Then by linearity of T, we have
T(x) = T(x1e1 + x2e2 + · · · + xnen)
= x1T(e1) + x2T(e2) + · · · + xnT(en)
= x1v1 + x2v2 + · · · + xnvn
= v1 v2 · · · vn x.
m×n 1
Define the matrix A ∈ M by A = v v2 · · · vn . Then our computation above
shows that
T(x) = x1v1 + x2v2 + · · · + xnvn = Ax.
Therefore, T is a matrix mapping with the matrix A ∈ Mm×n.
71
72
Onto, One-to-One, and Standard Matrix
If T : Rn → Rm is a linear mapping, the matrix
A = T(e
1
) T(e2) · · · T(en)
is called the standard matrix of T. In words, the columns of A are the images of the standard unit vectors e1, e2, . . . , en under T. The punchline is that if T is a linear mapping, then to derive properties of T we need only know the standard matrix A corresponding to T.
2 2
Example 8.15. Let T : R → R be the linear mapping that rotates every vector by an
1
0
and e2 =
0
1
angle θ. Use the standard unit vectors e1 = matrix A ∈ R2×2 corresponding to T.
in R2 to write down the
b b Tθ (e1)
θ
e1
b e2
Tθ (e2)
Solution. We have
A = T(e1) T(e2) =
"
cos(θ)
— sin(θ)
cos(θ)
sin(θ)
#
Example 8.16. Let T : R3 → R3 be a dilation of factor k = 2. Find the standard matrix
A of T.
Solution. The mapping is T(x) = 2x. Then
1 2
0
0
T(e ) = 2 0 = 0 , T(e
1 2
0 0
0
0
) = 2 1 = 2 , T(e
3
0 0
1
) = 2 0 = 0
2
Therefore,
2 0
A = T(e1) T(e2) T(e3) = 0
0
2 0
0 0 2
is the standard matrix of T.
After this lecture you should know the following:
73
Lecture 8
74
Onto, One-to-One, and Standard Matrix
Lecture 9
75
Lecture 9
Matrix Algebra
9.1 Sums of Matrices
We begin with the definition of matrix addition.
Definition 9.1: Given matrices
A =
a
· · · a
1n
a
11 a12
21 a22
· · · a
2n
.
.
. . .
. . .
, B =
b
· · · b
1n
b
11 b12
21 b22
· · · b
2n
. . . .
. . . .
am1 am2 amn bm1 bm2
· · · · · ·
bmn
,
both of the same dimension m × n, the sum A + B is defined as
A + B =
a + b · · · a + b
1n 1n
a + b
11 11 a12 + b12
21 21 a22 + b22
· · · a + b
2n 2n
.
.
. .
. .
.
.
am1 + bm1 am2 + bm2
· · ·
amn + bmn
.
Next is the definition of scalar-matrix multiplication.
Definition 9.2: For a scalar α we define αA by
αA = α
a · · · a
1n
a
11 a12
21 a22
· · · a
2n
.
.
. . .
. . .
am1 am2
· · ·
amn
=
11
αa αa12 · · · αa
1n
αa
21
αa22 · · · αa
2n
.
.
. .
.
.
αam1
. .
αam2 · · ·
αamn
.
76
Matrix Algebra
Example 9.3. Given A and B below, find 3A − 2B.
−2
A = 0 −3
1 5 5
0 −11
9 , B = 3 −5 1
4 −6 7 −1 −9 0
Solution. We compute:
3
3A − 2B = 0
27 − 6
2
21 −2
−6 15 10 0 −22
−9 −10
12 −18 −18
0
−7
= −6 1 25
14 0 21
−6 37
Below are some basic algebraic properties of matrix addition/scalar multiplication.
Theorem 9.4: Let A, B, C be matrices of the same size and let α, β be scalars. Then
9.2 Matrix Multiplication
Let TB : Rp → Rn and let TA : Rn → Rm be linear mappings. If x ∈ Rp then TB(x) ∈ Rn and thus we can apply TA to TB(x). The resulting vector TA(TB(x)) is in Rm. Hence, each x ∈ Rp can be mapped to a point in Rm, and because TB and TA are linear mappings the resulting mapping is also linear. This resulting mapping is called the composition of TA and TB, and is usually denoted by TA ◦ TB : Rp → Rm (see Figure 9.1). Hence,
(TA ◦ TB)(x) = TA(TB(x)).
Because (TA ◦ TB) : Rp → Rm is a linear mapping it has an associated standard matrix, which we denote for now by C. From Lecture 8, to compute the standard matrix of any linear mapping, we must compute the images of the standard unit vectors e1, e2, . . . , ep under the linear mapping. Now, for any x ∈ Rp,
TA(TB(x)) = TA(Bx) = A(Bx).
Applying this to x = ei for all i = 1, 2, . . . , p, we obtain the standard matrix of TA ◦ TB:
C = A(Be1) A(Be2) · · · A(Bep) .
77
Lecture 9
Rp
Rn
Rm
x
T (x)
TA(TB(x))
b B
TB
TA
(TA ◦ TB)(x)
Figure 9.1: Illustration of the composition of two mappings.
Now Be1 is
Be1 = b1 b2 · · · bp e1 = b1.
And similarly Bei = bi for all i = 1, 2, . . . , p. Therefore,
C = Ab1 Ab2 · · · Abp
is the standard matrix of TA ◦ TB. This computation motivates the following definition.
m×n n×p
1
Definition 9.5: For A ∈ R and B ∈ R , with B = b b2 · · · bp , we define the
product AB by the formula
AB = Ab1 Ab2 · · · Abp .
The product AB is defined only when the number of columns of A equals the number of rows of B. The following diagram is useful for remembering this:
(m × n) · (n × p) → m × p
From our definition of AB, the standard matrix of the composite mapping TA ◦ TB is
C = AB.
In other words, composition of linear mappings corresponds to matrix multiplication.
Example 9.6. For A and B below compute AB and BA.
A =
1 2 −2
1 1 −3
−4 2 4 −4
, B = −1 −5 −3 3
−4 −4 −3 −1
78
Matrix Algebra
Solution. First AB = [Ab1 Ab2 Ab3 Ab4]:
AB =
1 2 −2
1 1 −3
−1 −5
−4 −4
−4 2 4 −4
−3 3
−3 −1
=
2
7
2 0
7 9
2 0 4
7 9 10
2 0 4 4
7 9 10 2
=
=
=
On the other hand, BA is not defined! B has 4 columns and A has 2 rows.
Example 9.7. For A and B below compute AB and BA.
A = 3
−2 −1 1
3
−4 4 −1 −1
−3 −1 , B = −
0
3 0 −2
−2 1 −2
Solution. First AB = [Ab1 Ab2 Ab3]:
AB =
−4 4 3 −1 −1
0
−3 0 −2
−2 1 −2
3 −3 −1
−2 −1 1
−14
=
8
3
=
−14 7
8 −4
3 3
=
8 −4 8
3 3 0
−14 7 −14
79
Lecture 9
Next BA = [Ba1 Ba2 Ba3]:
BA = −
−1 −1 0
3 0 −2
−2 1 −2
−4 4
3
3 −3 −1
−2 −1 1
1
= 16
15
1 −1
= 16 −10
15 −9
1 −1
= 16 −10 −11
15 −9 −9
−2
On the other hand:
AB = 8 −4 8
3 3 0
−14 7 −14
Therefore, in general AB /= BA, i.e., matrix multiplication is not commutative.
An important matrix that arises frequently is the identity matrix In ∈ Rn×n of size
n:
n
I =
0 0 · · ·
0 1 0 · · · 0
.. .. .. · · · ..
1 0
0 0 0 · · · 1
You should verify that for any A ∈ Rn×n it holds that AIn = InA = A. Below are some basic algebraic properties of matrix multiplication.
Theorem 9.8: Let A, B, C be matrices, of appropriate dimensions, and let α be a scalar.
Then
If A ∈ Rn×n is a square matrix, the kth power of A is
k
A = AAA · · · A
` ˛¸ x
k times
80
Matrix Algebra
Example 9.9. Compute A3 if
A =
−2 3
1 0
.
Solution. Compute A2:
2
A =
−2 3 −2 3
1 0 1 0
=
7 −6
−2 3
And then A3:
A3 = A2A =
7 −6 −2 3
−2 3 1 0
−20 21
7 −6
=
We could also do:
A3 = AA2 =
−2 3 7 −6
1 0 −2 3
=
−20 21
7 −6
.
9.3 Matrix Transpose
We begin with the definition of the transpose of a matrix.
Definition 9.10: Given a matrix A ∈ Rm×n, the transpose of A is the matrix AT whose
ith column is the ith row of A.
If A is m × n then AT is n × m. For example, if
A =
0 −1 8 −7 −4
−4 6 −10 −9 6
9 5 −2 −3 5
−8 8 4 7 7
then
T
A =
8 −10
0 −4 9 −8
−1 6 5 8
−2
4
−7 −9 −3 7
−4 6 5 7
.
Example 9.11. Compute (AB)T and BT AT if
A =
−2 1 0
3 −1 −3
, B =
−2 1
2
−1 −2 0
0 0 −1
.
81
Lecture 9
Solution. Compute AB:
AB =
−2 1 0
3 −1 −3
3 −4 −4
−5 5 9
−2 1
2
−1 −2 0
0 0 −1
=
Next compute BT AT :
−2
3
1 −1
0 −3
−2 −1 0
BT AT = 1 −2 0
2 0 −1
3
= −4
−5
5 = (AB)
T
−4 9
The following theorem summarizes properties of the transpose.
Theorem 9.12: Let A and B be matrices of appropriate sizes. The following hold:
A consequence of property (4) is that
(A1A2 . . . Ak)T = AT AT
k k−1
· · · A A
T T
2 1
and as a special case
(Ak)T = (AT )k.
Example 9.13. Let T : R2 → R2 be the linear mapping that first contracts vectors by a factor of k = 3 and then rotates by an angle θ. What is the standard matrix A of T?
Solution. Let e1 = (1, 0) and e2 = (0, 1) denote the standard unit vectors in R2. From
1
Lecture 8, the standard matrix of T is A = T(e ) T(e2) . Recall that the standard matrix
of a rotation by θ is
cos(θ) − sin(θ) sin(θ) cos(θ)
3
Contracting e1 by a factor of k = 3 results in (1 , 0) and then rotation by θ results in
1
3
cos(θ)
3
1 sin(θ)
= T(e1).
82
Matrix Algebra
3
Contracting e2 by a factor of k = 3 results in (0, 1 ) and then rotation by θ results in
1
3
— sin(θ)
3
1 cos(θ)
= T(e2).
Therefore,
1
2
A = T(e ) T(e ) =
"
1
3
1
3
cos(θ) − sin(θ)
1 sin(θ) 1 cos(θ)
3 3
#
3
On the other hand, the standard matrix corresponding to a contraction by a factor k = 1 is
1
3
0
1
3
" #
0
Therefore,
"cos(θ) − sin(θ)
sin(θ) cos(θ) 0
`
1
3
0
1
3
rot˛a¸tion x c`ontr˛a¸ctioxn
# " # "
=
1
3
1
3
cos(θ) − sin(θ)
3
1 sin(θ)
3
1 cos(θ)
#
= A
After this lecture you should know the following:
Lecture 10
83
Lecture 10
Invertible Matrices
10.1 Inverse of a Matrix
The inverse of a square matrix A ∈ Rn×n generalizes the notion of the reciprocal of a non- zero number a ∈ R. Formally speaking, the inverse of a non-zero number a ∈ R is the unique
number c ∈ R such that ac = ca = 1. The inverse of a /= 0, usually denoted by a
−1
1
a
= , can
be used to solve the equation ax = b:
ax = b ⇒ a−1ax = a−1b ⇒ x = a−1b.
This motivates the following definition.
Definition 10.1: A matrix A ∈ Rn×n is called invertible if there exists a matrix C ∈
Rn×n such that AC = In and CA = In.
If A is invertible then can it have more than one inverse? Suppose that there exists C1, C2
such that ACi = CiA = In. Then
C2 = C2(AC1) = (C2A)C1 = InC1 = C1.
Thus, if A is invertible, it can have only one inverse. This motivates the following definition.
Definition 10.2: If A is invertible then we denote the inverse of A by A−1. Thus,
AA−1 = A−1A = In.
Example 10.3. Given A and C below, show that C is the inverse of A.
A =
−2 6 1
0
−1 2 −2 , C =
−5 −1 −2
2 0 1
1 −3 −14 −3 −6
84
Invertible Matrices
Solution. Compute AC:
AC = −1 2 −2
−2 6 1
1 −3 0 −14
−3 −6 1 0
−5 −1 −2 = 0
0
1 0
2 0 1 0 0 1
Compute CA:
−14 −3
CA = −5 −1 −2
2 0 1
−6
0 1
0
1 −3
−1 2 −2 = 0 1 0
−2 6 1 0 0 1
0
Therefore, by definition C = A−1.
Theorem 10.4: Let A ∈ Rn×n and suppose that A is invertible. Then for any b ∈ Rn
the matrix equation Ax = b has a unique solution given by A−1b.
Proof: Let b ∈ Rn be arbitrary. Then multiplying the equation Ax = b by A−1 from the left we obtain that
A−1Ax = A−1b
−1
⇒ Inx = A b
−1
⇒ x = A b.
Therefore, with x = A−1b we have that
Ax = A(A−1b) = AA−1b = Inb = b
and thus x = A−1b is a solution. If x˜ is another solution of the equation, that is, Ax˜ = b, then multiplying both sides by A−1 we obtain that x˜ = A−1b. Thus, x = x˜. Example 10.5. Use the result of Example 10.3. to solve the linear system Ax = b if
2
0
1
−1
1
1 −3
A = −1 2 −
0
2 , b = −
−1
1
3 .
−2 6 1
Solution. We showed in Example 10.3 that
−1
−14 −3 −6
A = −5 −1 −2 .
2 0 1
Therefore, the unique solution to the linear system Ax = b is
−1
A b = −5
−6 1 1
−14 −3
−1 −2 −3 = 0
85
Lecture 10
Verify:
1 −3
0 1 1
−1 2 −
−2 6 1 1
2 0 = −3
−1
The following theorem summarizes the relationship between the matrix inverse and ma- trix multiplication and matrix transpose.
Theorem 10.6: Let A and B be invertible matrices. Then:
(A−1)−1 = A.
(AB)−1 = B−1A−1.
(AT )−1 = (A−1)T .
Proof: To prove (2) we compute
(AB)(B−1A−1) = ABB−1A−1 = AInA−1 = AA−1 = In.
To prove (3) we compute
n
AT (A−1)T = (A−1A)T = IT = In.
10.2 Computing the Inverse of a Matrix
If A ∈ M
n×n
−1
is invertible, how do we find A ?
−1
Let A = c
c · · ·
1 2 n
c and we will
i
−1
find expressions for c . First note that AA = Ac1 Ac2 · · · Acn . On the other hand,
n 1 2 n
we also have AA−1 = In = e1 e2 · · · e . Therefore, we want to find c , c , . . . , c such
that
` ˛¸
−1
x ` ˛¸ x
AA In
Ac1 Ac2 · · · Acn = e1 e2 · · · en .
To find ci we therefore need to solve the linear system Ax = ei. Here the image vector “b” is ei. To find c1 we form the augmented matrix A e1 and find its RREF:
A
e1 ∼ In c1 .
86
Invertible Matrices
We will need to do this for each c2, . . . , cn so we might as well form the combined augmented matrix A e1 e2 · · · en and find the RREF all at once:
A
e1 e2 · · · en ∼ In c1 c2 · · · cn .
In summary, to determine if A−1 exists and to simultaneously compute it, we compute the RREF of the augmented matrix
A
In ,
that is, A augmented with the n × n identity matrix. If the RREF of A is In, that is
In ∼ In c1 c2 · · ·
A cn
then
−1
A = c
1
c2 · · · cn .
If the RREF of A is not In then A is not invertible.
Example 10.7. Find the inverse of A = 1 3 if it exists.
−1 −2
Solution. Form the augmented matrix A I2 and row reduce:
2
A
I =
1 3 1 0
−1 −2 0 1
Add rows R1 and R2:
1 3 1 0
−1 −2 0 1
R1 +R2
−−−−→
1 3 1 0
0 1 1 1
−3R2 +R1
Perform the operation −−−−−→ :
−−−−−→
−3R2 +R1 1
1 3 1 0 0 −2 −3
0 1 1 1 0 1 1 1
Thus, rref(A) = I2, and therefore A is invertible. The inverse is
−1
A =
−2 −3
1 1
Verify:
−1
AA =
1 3 −2 −3
−1 −2 1 1
=
1 0
0 1
.
1 0
Example 10.8. Find the inverse of A = 1 1 0
−2 0 −7
3
if it exists.
87
Lecture 10
Solution. Form the augmented matrix A
3
I and row reduce:
1 0
−R1 +R2, 2R1 +R2
1
0
−R3:
1
3 1
0 1 −3 −1
0 0 −1 2
−R3
0 1 0 3
0 −−→ 0 1 −3
1 0 0 1
0 3 1 0 0 3 1 0
1 1 0 0 1 0 −−−−−−−−−−→ 0 1 −3 −1 1 0
−2 0 −7 0 0 1 0 0 −1 2 0 1
0 0 1
1 −1
0 −2
0 0
1 0
0 −1
3R3 + R2 and −3R3 + R1:
1
0 3 1 0
0
0 1 −3 −1 1 0
3R3 +R2 , −3R3 +R1
−−−−−−−−−−−→
1
0 0 1 −2 0 −1 0
0 0 7 0
0 1 0 −7 1 −3
0 1 −2 0 −1
3
Therefore, rref(A) = I3, and therefore A is invertible. The inverse is
−1
0
A = −7 1 −3
−2 0 −1
7 3
Verify:
−1
1
0
AA = 1
3 7
3 1
0
−3 = 0
1 0
0
0
1 0 −7 1
−2 0 −7 −2 0 −1 0 0 1
1
Example 10.9. Find the inverse of A =
1 0
1 1 −2 if it exists.
−2 0 −2
3
Solution. Form the augmented matrix A I and row reduce:
1 0
−R1 +R2, 2R1 +R2
1
0 1 1 0
1 1 −2 0 1 0 −−−−−−−−−−→ 0
−2 0 −2 0
0 1 1 0
1 −3 −1 1 0
0 1 0 0 0 2 0 1
0
We need not go further since the rref(A) is not I3 (rank(A) = 2 ). invertible.
10.3 Invertible Linear Mappings
Therefore, A is not
Let TA : Rn → Rn be a matrix mapping with standard matrix A and suppose that A is invertible. Let TA−1 : Rn → Rn be the matrix mapping with standard matrix A−1. Then the standard matrix of the composite mapping TA−1 ◦ TA : Rn → Rn is
A−1A = In.
88
Invertible Matrices
Therefore, (TA−1 ◦ TA)(x) = Inx = x. Let’s unravel (TA−1 ◦ TA)(x) to see this: (TA−1 ◦ TA)(x) = TA−1 (TA(x)) = TA−1 (Ax) = A−1Ax = x.
Similarly, the standard matrix of (TA ◦TA−1 ) is also In. Intuitively, the linear mapping TA−1 undoes what TA does, and conversely. Moreover, since Ax = b always has a solution, TA is onto. And, because the solution to Ax = b is unique, TA is one-to-one.
The following theorem summarizes equivalent conditions for matrix invertibility.
Theorem 10.10: Let A ∈ Rn×n. The following statements are equivalent:
Proof: This is a summary of all the statements we have proved about matrices and matrix mappings specialized to the case of square matrices A ∈ Rn×n. Note that for non-square matrices, one-to-one does not imply ontoness, and conversely.
Example 10.11. Without doing any arithmetic, write down the inverse of the dilation matrix
3 0
" #
A = .
0 5
Example 10.12.
matrix
Without doing any arithmetic, write down the inverse of the rotation
A =
cos(θ)
— sin(θ)
cos(θ)
" #
sin(θ)
.
After this lecture you should know the following:
Lecture 11
89
Lecture 11
Determinants
11.1 Determinants of 2 × 2 and 3 × 3 Matrices
Consider a general 2 × 2 linear system
a11x1 + a12x2 = b1 a21x1 + a22x2 = b2.
Using elementary row operations, it can be shown that the solution is
1
b1a22 − b2a12
x = ,
a11a22 − a12a21
2
b2a11 − b1a21
x = ,
a11a22 − a12a21
provided that a11a22 − a12a21 /= 0. Notice the denominator is the same in both expressions. The number a11a22 − a12a21 then completely characterizes when a 2 × 2 linear system has a unique solution. This motivates the following definition.
Definition 11.1: Given a 2 × 2 matrix
A =
11
a a
12
a21 a22
we define the determinant of A as
det A = det
11
a a
12
a21 a22
11 22 12 21
= a a − a a .
An alternative notation for det A is using vertical bars:
det
11
a a
12
a21 a22
=
a a
11 12
a a
21 22
.
90
Determinants
Example 11.2. Compute the determinant of A.
(i) A =
3 −1
8 2
(ii) A =
3 1
−6 −2
(iii) A =
−110 0
568 0
Solution. For (i):
det(A) =
3 −1
8 2
= (3)(2) − (8)(−
1) = 14
For (ii):
det(A) =
3 1
−6 −2
= (3)(−
2) − (−
6)(1) = 0
For (iii):
det(A) =
−110 0
568 0
= (−110)(0) − (568)(0) = 0
As in the 2 × 2 case, the solution of a 3 × 3 linear system Ax = b can be shown to be
x1 =
Numerator1 Numerator2
, x2 = , x3 =
Numerator3
D D D
where
D = a11(a22a33 − a23a32) − a12(a21a33 − a23a31) + a13(a21a32 − a22a31).
Notice that the terms of D in the parenthesis are determinants of 2 × 2 submatrices of A:
22 23
a a
a32 a33
a a
21 23
a31 a33
a a
21 22
a31 a32
D = a11(a22a33 − a23a32) − a12(a21a33 − a23a31) + a13(a21a32 − a22a31).
Let
11
A =
22
a a
23
a32 a33
` ˛¸ x ` ˛¸ x ` ˛¸ x
, A
12
=
21
a a
23
a31 a33
13
, and A =
21
a a
22
a31 a32
.
Then we can write
D = a11 det(A11) − a12 det(A12) + a13 det(A13).
11
The matrix A =
a a
22 23
a32 a33
is obtained from A by deleting the 1st row and the 1st column:
A = a a
21 a22 23
a31 a32 a33
a11 a12 a13
11
−→ A =
22
a a
23
a32 a33
.
91
Lecture 11
12
Similarly, the matrix A = 2nd column:
21
a a
23
a31 a33
is obtained from A by deleting the 1st row and the
A = a
21 a22 23
a31 a32 a33
a11 a12 a13
12
a −→ A =
21
a a
23
a31 a33
.
13
Finally, the matrix A = column:
21
a a
22
a31 a32
is obtained from A by deleting the 1st row and the 3rd
a
13
A = a
a11 a12
21 a22 23
a31 a32 a33
a −→
21
a a
22
a31 a32
.
Notice also that the sign in front of the coefficients a11, a12, and a13, alternate. This motivates the following definition.
Definition 11.3: Let A be a 3 × 3 matrix. Let Ajk be the 2 × 2 matrix obtained from A by deleting the jth row and kth column. Define the cofactor of ajk to be the number Cjk = (−1)j+k det Ajk. Define the determinant of A to be
det A = a11C11 + a12C12 + a13C13.
This definition of the determinant is called the expansion of the determinant along the first row. In the cofactor Cjk = (−1)j+k det Ajk, the expression (−1)j+k will evaluate to either 1 or −1, depending on whether j + k is even or odd. For example, the cofactor of a12 is
1+2
C12 = (−1) det A12 = − det A12
1+3
C13 = (−1) det A13 = det A13.
and the cofactor of a13 is
We can also compute the cofactor of the other entries of A in the obvious way. For example, the cofactor of a23 is
2+3
C23 = (−1) det A23 = − det A23.
A helpful way to remember the sign (−1)j+k of a cofactor is to use the matrix
—
+ +
− + − .
+ − +
This works not just for 3 × 3 matrices but for any square n × n matrix.
Example 11.4. Compute the determinant of the matrix
−2
A = 2
3 5
1 0 6
4 3
92
Determinants
Solution. From the definition of the determinant
det A = a11C11 + a12C12 + a13C13
= (4) det A11 − (−2) det A12 + (3) det A13
= 4
3 5 2 5
+ 2 + 3
2 3
0 6 1 6 1 0
= 4(3 · 6 − 5 · 0) + 2(2 · 6 − 1 · 5) + 3(2 · 0 − 1 · 3)
= 72 + 14 − 9
= 77
We can compute the determinant of a matrix A by expanding along any row or column.
For example, the expansion of the determinant for the matrix
a11
A = a
a
a12 a13
21 a22 23
a31 a32 a33
along the 3rd row is
det A = a
31
12
a a
13
22
a a
23
— a
32
11
a a
13
21
a a
23
+ a
33
11
a a
12
21
a a
22
.
And along the 2nd column:
det A = −a
12
21
a a
23
+ a
22
11
a a
13
a a a
— a
32
11
a a
13
a a a
31 33 31 33 21 23
.
The punchline is that any way you choose to expand (row or column) you will get the same answer. If a particular row or column contains zeros, say entry ajk, then the computation of the determinant is simplified if you expand along either row j or column k because ajkCjk = 0 and we need not compute Cjk.
Example 11.5. Compute the determinant of the matrix
−2
A = 2 3 5
4 3
1 0 6
Solution. In Example 11.4, we computed det(A) = 77 by expanding along the 1st row.
93
Lecture 11
Notice that a32 = 0. Expanding along the 3rd row:
det A = (1) det A31 − (0) det A32 + (6) det A33
=
−2 3
+ 6
4 −2
3 5 2 3
= 1(−2 · 5 − 3 · 3) + 6(4 · 3 − (−2) · 2)
= −19 + 96
= 77
11.2 Determinants of n × n Matrices
Using the 3 × 3 case as a guide, we define the determinant of a general n × n matrix as follows.
Definition 11.6: Let A be a n × n matrix. Let Ajk be the (n − 1) × (n − 1) matrix
j+k
obtained from A by deleting the jth row and kth column, and let Cjk = (−1) det Ajk
be the (j, k)-cofactor of A. The determinant of A is defined to be
det A = a11C11 + a12C12 + · · · + a1nC1n.
The next theorem tells us that we can compute the determinant by expanding along any row or column.
Theorem 11.7: Let A be a n × n matrix. Then det A may be obtained by a cofactor expansion along any row or any column of A:
det A = aj1Cj1 + aj2Cj2 + · · · + ajnCjn.
We obtain two immediate corollaries.
Corollary 11.8: If A has a row or column containing all zeros then det A = 0.
Proof. If the jth row contains all zeros then aj1 = aj2 = · · · = ajn = 0:
det A = aj1Cj1 + aj2Cj2 + · · · + ajnCjn = 0.
94
Determinants
Corollary 11.9: For any square matrix A it holds that det A = det AT .
Sketch of the proof. Expanding along the jth row of A is equivalent to expanding along the jth column of AT .
Example 11.10. Compute the determinant of
A =
1 | 3 |
1 | 2 |
0 | 0 |
−1 | −3 |
0 −2
−2 −1
2 1
1 0
Solution. The third row contains two zeros, so expand along this row:
det A = 0 det A31 − 0 det A32 + 2 det A33 − det A34
1 3 −2
−1 −3 0
= 2 1 2 −1 −
1 3 0
1 2 −2
−1 −3 1
= 2 1
2 −1
−3 0
— 3
1 −1
−1 0
— 2
1 2
−1 −3
— 1
2 −2
— 3
1 −2
−3 1 −1 1
= 2((0 − 3) − 3(0 − 1) − 2(−3 + 2)) − ((2 − 6) − 3(1 − 2))
= 5
Example 11.11. Compute the determinant of
A =
1 | 3 |
1 | 2 |
0 | 0 |
−1 | −3 |
0 −2
−2 −1
2 1
1 0
95
Lecture 11
Solution. Expanding along the second row:
det A = − det A21 + 2 det A22 − (−2) det A23 − 1 det A24
= −
3 0 −2 1 0 −2
0 2 1 + 2 0 2 1
−3 1 0 −1 1 0
+ 2
0 1 −
1 3 −2 1 3 0
0 0 0 2
−1 −3 0 −1 −3 1
= −1(−3 − 12) + 2(−1 − 4) + 2(0) − (0)
= 5
11.3 Triangular Matrices
Below we introduce a class of matrices for which the determinant computation is trivial.
Definition 11.12: A square matrix A ∈ Rn×n is called upper triangular if ajk = 0 whenever j > k. In other words, all the entries of A below the diagonal entries aii are zero. It is called lower triangular if ajk = 0 whenever j < k.
A = 0
For example, a 4 × 4 upper triangular matrix takes the form a11 a12 a13 a14 0 a22 a23 a24 | |||
| 0 | a33 | |
0 | 0 | 0 | a44 |
a34
Expanding along the first column, we compute
det A = a
11
a a
34
22 a23 24
0 a33 a
0 0
a44
= a
11 a22
0
a44
a33 a34
= a11a22a33a44.
The general n × n case is similar and is summarized in the following theorem.
Theorem 11.13: The determinant of a triangular matrix is the product of its diagonal entries.
After this lecture you should know the following:
T
96
Determinants
Lecture 12
97
Lecture 12
Properties of the Determinant
12.1 ERO and Determinants
Recall that for a matrix A ∈ Rn×n we defined
det A = aj1Cj1 + aj2Cj2 + · · · + ajnCjn
where the number Cjk = (−1)j+k det Ajk is called the (j, k)-cofactor of A and
j
a = a
· · ·
j1 aj2 ajn
denotes the jth row of A. Notice that
j1
j2
det A = a a · · · a
jn
C
Cj1
j2
.
.
Cjn
.
If we let cj = Cj1
Cj2
· · · Cjn then
j
T
j
det A = a · c .
In this lecture, we will establish properties of the determinant under elementary row opera- tions and some consequences. The following theorem describes how the determinant behaves under elementary row operations of Type 1.
Theorem 12.1: Suppose that A ∈ Rn×n and let B be the matrix obtained by interchang- ing two rows of A. Then det B = − det A.
Proof. Consider the 2 × 2 case. Let A =
a a
11 12
a21 a22
a21 a22 a11 a12
and let B = . Then
det B = a12a21 − a11a22 = −(a11a22 − a12a21) = − det A.
The general case is proved by induction.
This theorem leads to the following corollary.
98
Properties of the Determinant
Corollary 12.2: If A ∈ Rn×n has two rows (or two columns) that are equal then det(A) = 0.
Proof. Suppose that A has rows j and k that are equal. Let B be the matrix obtained by interchanging rows j and k. Then by the previous theorem det B = − det A. But clearly B = A, and therefore det B = det A. Therefore, det(A) = − det(A) and thus det A = 0.
Now we consider how the determinant behaves under elementary row operations of Type
2.
Theorem 12.3: Let A ∈ Rn×n and let B be the matrix obtained by multiplying a row of
A by β. Then det B = β det A.
Proof. Suppose that B is obtained from A by multiplying the jth row by β. The rows of A
and B different from j are equal, and therefore
Bjk = Ajk, for k = 1, 2, . . . , n.
In particular, the (j, k) cofactors of A and B are equal. The jth row of B is βaj . Then, expanding det B along the jth row:
j
det B = (βa ) · c
T
j
j
T
j
= β(a · c )
= β det A.
Lastly we consider Type 3 elementary row operations.
Theorem 12.4: Let A ∈ Rn×n and let B be the matrix obtained from A by adding β
times the kth row to the jth row. Then det B = det A.
Proof. For any matrix A and any row vector r = [r1 r2 · · · rn] the expression
T
j
r · c = r C + r C + · · · + r C
1 j1 2 j2 n jn
is the determinant of the matrix obtained from A by replacing the jth row with the row r. Therefore, if k /= j then
k
T
j
a · c = 0
99
Lecture 12
since then rows k and j are equal. The jth row of B is bj = aj + βak. Therefore, expanding det B along the jth row:
det B = (aj + βak) · c
T
j
T
j
j k
= a · c + β a ·
c
T
j
= det A.
Example 12.5. Suppose that A is a 4 × 4 matrix and suppose that det A = 11. If B is obtained from A by interchanging rows 2 and 4, what is det B?
Solution. Interchanging (or swapping) rows changes the sign of the determinant. Therefore, det B = −11.
Example 12.6. Suppose that A is a 4 × 4 matrix and suppose that det A = 11. Let a1, a2, a3, a4 denote the rows of A. If B is obtained from A by replacing row a3 by 3a1 + a3, what is det B?
Solution. This is a Type 3 elementary row operation, which preserves the value of the de- terminant. Therefore,
det B = 11.
Example 12.7. Suppose that A is a 4 × 4 matrix and suppose that det A = 11. Let a1, a2, a3, a4 denote the rows of A. If B is obtained from A by replacing row a3 by 3a1 + 7a3, what is det B?
Solution. This is not quite a Type 3 elementary row operation because a3 is multiplied by
7. The third row of B is b3 = 3a1 + 7a3. Therefore, expanding det B along the third row
det B = (3a1 + 7a3) · c
T
3
T
3
1 3
= 3a · c + 7a · c
T
3
3
T
3
= 7(a · c )
= 7 det A
= 77
100
Properties of the Determinant
Example 12.8. Suppose that A is a 4 × 4 matrix and suppose that det A = 11. Let a1, a2, a3, a4 denote the rows of A. If B is obtained from A by replacing row a3 by 4a1 + 5a2, what is det B?
Solution. Again, this is not a Type 3 elementary row operation. The third row of B is
b3 = 4a1 + 5a2. Therefore, expanding det B along the third row
det B = (4a1 + 5a2) · c
T
3
T
3
1 2
= 4a · c + 5a · c
T
3
= 0 + 0
= 0
The following theorem characterizes invertibility of matrices with the determinant.
Theorem 12.9: A square matrix A is invertible if and only if det A /= 0.
Proof. Beginning with the matrix A, perform elementary row operations and generate a sequence of matrices A1, A2, . . . , Ap such that Ap is in row echelon form and thus triangular:
A ~ A1 ~ A2 ~ · · · ~ Ap.
Thus, matrix Ai is obtained from Ai−1 by performing one of the elementary row operations. From Theorems 12.1, 12.3, 12.4, if det Ai−1 /= 0 then det Ai /= 0. In particular, det A = 0 if and only if det Ap = 0. Now, Ap is triangular and therefore its determinant is the product of its diagonal entries. If all the diagonal entries are non-zero then det A = det Ap /= 0. In this case, A is invertible because there are r = n leading entries in Ap. If a diagonal entry of Ap is zero then det A = det Ap = 0. In this case, A is not invertible because there are r < n leading entries in Ap. Therefore, A is invertible if and only if det A /= 0.
The following theorem characterizes how the determinant behaves under scalar multiplication of matrices.
Theorem 12.10: Let A ∈ Rn×n and let B = βA, that is, B is obtained by multiplying every entry of A by β. Then det B = βn det A.
101
Lecture 12
Proof. Consider the 2 × 2 case:
det(βA) =
βa βa
11 12
12
βa βa22
= βa11 · βa22 − βa12 · βa21
= β2(a11a22 − a12a21)
= β2 det A.
Thus, the statement holds for 2 × 2 matrices. Consider a 3 × 3 matrix A. Then det(βA) = βa11|βA11| − βa12|βA12| + βa13|βA13|
= βa11β2|A11| − βa12β2|A12| + βa13β2|A13|
= β3 (a11|A11| − a12|A12| + a13|A13|)
= β3 det A.
The general case can be treated using mathematical induction on n.
Example 12.11. Suppose that A is a 4 × 4 matrix and suppose that det A = 11. What is det(3A)?
Solution. We have
det(3A) = 34 det A
= 81 · 11
= 891
The following theorem characterizes how the determinant behaves under matrix multi- plication.
Theorem 12.12: Let A and B be n × n matrices. Then
det(AB) = det(A) det(B).
Corollary 12.13: For any square matrix det(Ak) = (det A)k.
102
Properties of the Determinant
Corollary 12.14: If A is invertible then
det(A−1) =
1
det A
.
Proof. From AA−1 = In we have that det(AA−1) = 1. But also
det(AA−1) = det(A) det(A−1).
Therefore
det(A) det(A−1) = 1
or equivalently
det A−1 =
1
det A
.
Example 12.15. Let A, B, C be n × n matrices. Suppose that det A = 3, det B = 0, and det C = 7.
Solution. (i): We have det(AC) = det A det C = 3 · 7 = 21. Thus, AC is invertible.
After this lecture you should know the following:
Lecture 13
103
Lecture 13
Applications of the Determinant
13.1 The Cofactor Method
Recall that for A ∈ Rn×n we defined
det A = aj1Cj1 + aj2Cj2 + · · · + ajnCjn
where Cjk = (−1)j+k det Ajk is called the (j, k)-Cofactor of A and
aj = aj1 · · ·
aj2 ajn
is the jth row of A. If cj = Cj1
Cj2
· · · Cjn then
j1 j2
det A = a a · · · a
jn
C
j2
.
.
Cj1
Cjn
Suppose that B is the matrix obtained from A by replacing row aj with a distinct row ak. To compute det B expand along its jth row bj = ak:
j
T
j
= a · c .
k
T
j
det B = a · c = 0.
j
The Cofactor Method is an alternative method to find the inverse of an invertible matrix.
Recall that for any matrix A ∈ Rn×n, if we expand along the jth row then
T
j
det A = a · c .
On the other hand, if j /= k then
In summary,
j
T
k
a · c = 0.
j
T
k
a · c =
(
det A, if j = k
0, if j /= k.
104
Applications of the Determinant
Form the Cofactor matrix
Cof(A) =
C
C
11 C12
21 C22
2n
. .
Cn1 Cn2
Cnn
2
. .
· · ·
· · · C c
. . · · · . .
· · ·
cn
C1n c1
=
.
Then,
T
A(Cof(A)) =
a1
2
.
.
an
a
c
T cT
1 2
· · · c
T
n
=
1
a c
T
1
2
a1cT
2
1
T
n
a c
T
1
a2cT
2
2
· · · a c
· · · a c
T
n
.
.
.
.
. .
.
.
n
a c
T
1
2
n
T
n
.
=
det A
0
det A
0
.
.
.
.
.
ancT · · · a c
0 · · · 0
· · ·
. . .
0 · · ·
0
det A
.
This can be written succinctly as
A(Cof(A))T = det(A)In.
Now if det A /= 0 then we can divide by det A to obtain
A
1
det A
T
n
(Cof(A)) = I .
This leads to the following formula for the inverse:
A−1 =
1
det A
(Cof(A))T
Although this is an explicit and elegant formula for A−1, it is computationally intensive, even for 3 × 3 matrices. However, for the 2 × 2 case it provides a useful formula to compute
105
Lecture 13
the matrix inverse. Indeed, if A =
a b
c d
we have Cof(A) =
d −c
−b a
and therefore
−1
A =
ad − bc
−c a
1 d −b
.
When does an integer matrix have an integer inverse? We can answer this question using the Cofactor Method. Let us first be clear about what we mean by an integer matrix.
Definition 13.1: A matrix A ∈ Rm×n is called an integer matrix if every entry of A is an integer.
Suppose that A ∈ Rn×n is an invertible integer matrix. Then det(A) is a non-zero integer and (Cof(A))T is an integer matrix. If A−1 is also an integer matrix then det(A−1) is also an integer. Now det(A) det(A−1) = 1 thus it must be the case that det(A) = ±1. Suppose on the other hand that det(A) = ±1. Then by the Cofactor method
−1
A =
1 det(A)
T
(Cof(A)) = ±(Cof(A))
T
and therefore A−1 is also an integer matrix. We have proved the following.
Theorem 13.2: An invertible integer matrix A ∈ Rn×n has an integer inverse A−1 if and only if det A = ±1.
We can use the previous theorem to generate integer matrices with an integer inverse as follows. Begin with an upper triangular matrix M0 having integer entries and whose diagonal entries are either 1 or −1. By construction, det(M0) = ±1. Perform any sequence of elementary row operations of Type 1 and Type 3. This generates a sequence of matrices M1, . . . , Mp whose entries are integers. Moreover,
M0 ~ M1 ~ · · · ~ Mp.
Therefore,
±1 = det(M) = det(M1) = · · · = det(Mp).
106
Applications of the Determinant
13.2 Cramer’s Rule
The Cofactor method can be used to give an explicit formula for the solution of a linear system where the coefficient matrix is invertible. The formula is known as Cramer’s Rule. To derive this formula, recall that if A is invertible then the solution to Ax = b is x = A−1b.
det A
Using the Cofactor method, A−1 = 1 (Cof(A))T , and therefore
x =
1
det A
C · · ·
C
11 C21
12 C22
. .
. .
. .
.
C1n C2n
· · ·
Cnn
· · · C b
n2 2
. .
. .
bn
Cn1 b1
.
Consider the first component x1 of x:
1
x1 = (b1C11 + b2C21 + · · · + bnCn1).
det A
The expression b1C11 + b2C21 + · · · + bnCn1 is the expansion of the determinant along the
first column of the matrix obtained from A by replacing the first column with b:
det
1 a12
b · · · a
1n
b
2 a22
· · · a
2n
. .
. .
. .
.
.
.
bn
an2
· · ·
ann
= b1C11 + b2C21 + · · · + bnCn1
Similarly,
1
x2 = (b1C12 + b2C22 + · · · + bnCn2) det A
x =
1
det A
det A
2
.
.
det An
and (b1C12 + b2C22 + · · · + bnCn2) is the expansion of the determinant along the second
column of the matrix obtained from A by replacing the second column with b. In summary:
Theorem 13.3: (Cramer’s Rule) Let A ∈ Rn×n be an invertible matrix. Let b ∈ Rn
and let Ai be the matrix obtained from A by replacing the ith column with b. Then the solution to Ax = b is
det A1
.
Although this is an explicit and elegant formula for x, it is computationally intensive, and used mainly for theoretical purposes.
107
Lecture 13
13.3 Volumes
The volume of the parallelepiped determined by the vectors v1, v2, v3 is
T
1
1 2 3 2 3
1 3
Vol(v , v , v ) = abs(v (v × v )) = abs(det v v2 v )
where abs(x) denotes the absolute value of the number x. Let A be an invertible matrix and let w1 = Av1, w2 = Av2, w3 = Av3. How are Vol(v1, v2, v2) and Vol(w1, w2, w2) related? Compute:
Vol(w1, w2, w3) = abs(det w1 w2 w3 )
= abs det Av1 Av2
= abs det(A v1 v2
Av3
v3 )
= abs det A · det v1 v2
v3
= abs(det A) · Vol(v1, v2, v3).
Therefore, the number abs(det A) is the factor by which volume is changed under the linear transformation with matrix A. In summary:
Theorem 13.4: Suppose that v1, v2, v3 are vectors in R3 that determine a parallelepiped of non-zero volume. Let A be the matrix of a linear transformation and let w1, w2, w3 be the images of v1, v2, v3 under A, respectively. Then
Vol(w1, w2, w3) = abs(det A) · Vol(v1, v2, v3).
Example 13.5. Consider the data
4
A = 2 4 1
1 1 4
1 −1
1
, v =
1
, v = 1
2 3
−1
0 2 1
0
, v = 5 .
−1
and let w1 = Av1, w2 = Av2, and w3 = Av3. Find the volume of the parallelepiped spanned by the vectors {w1, w2, w3}.
Solution. We compute:
Vol(v1, v2, v3) = abs(det( v1 v2 v3 )) = abs(−7) = 7
We compute:
det(A) = 55.
Therefore, the volume of the parallelepiped spanned by the vectors {w1, w2, w3} is
Vol(w1, w2, w3) = abs(55) × 7 = 385.
108
Applications of the Determinant
After this lecture you should know the following:
Lecture 14
Vector Spaces
When you read/hear the word vector you may immediately think of two points in R2 (or R3) connected by an arrow. Mathematically speaking, a vector is just an element of a vector space. This then begs the question: What is a vector space? Roughly speaking, a vector space is a set of objects that can be added and multiplied by scalars. You have already worked with several types of vector spaces. Examples of vector spaces that you have already encountered are:
In all of these sets, there is an operation of “addition“ and “multiplication by scalars”. Let’s formalize then exactly what we mean by a vector space.
Definition 14.1: A vector space is a set V of objects, called vectors, on which two operations called addition and scalar multiplication have been defined satisfying the following properties. If u, v, w are in V and if α, β ∈ R are scalars:
110
Vector Spaces
It can be shown that 0 · v = 0 for any vector v in V. To better understand the definition of a vector space, we first consider a few elementary examples.
Example 14.2. Let V be the unit disc in R2:
V = {(x, y) ∈ R2 | x2 + y2 ≤ 1}
Is V a vector space?
Solution. The circle is not closed under scalar multiplication. For example, take u = (1, 0) ∈ V and multiply by say α = 2. Then αu = (2, 0) is not in V. Therefore, property (6) of the definition of a vector space fails, and consequently the unit disc is not a vector space.
Example 14.3. Let V be the graph of the quadratic function f(x) = x2:
V = n(x, y) ∈ R2 | y = x2,.
Is V a vector space?
Solution. The set V is not closed under scalar multiplication. For example, u = (1, 1) is a point in V but 2u = (2, 2) is not. You may also notice that V is not closed under addition either. For example, both u = (1, 1) and v = (2, 4) are in V but u + v = (3, 5) and (3, 5) is not a point on the parabola V. Therefore, the graph of f(x) = x2 is not a vector space.
111
Lecture 14
Example 14.4. Let V be the graph of the function f(x) = 2x:
V = {(x, y) ∈ R2 | y = 2x}.
Is V a vector space?
Solution. We will show that V is a vector space. First, we verify that V is closed under addition. We first note that an arbitrary point in V can be written as u = (x, 2x). Let then u = (a, 2a) and v = (b, 2b) be points in V. Then
u + v = (a + b, 2a + 2b) = (a + b, 2(a + b)).
Therefore V is closed under addition. Verify that V is closed under scalar multiplication:
αu = α(a, 2a) = (αa, α2a) = (αa, 2(αa)).
Therefore V is closed under scalar multiplication. There is a zero vector 0 = (0, 0) in V:
u + 0 = (a, 2a) + (0, 0) = (a, 2a).
All the other properties of a vector space can be verified to hold; for example, addition is commutative and associative in V because addition in R2 is commutative/associative, etc. Therefore, the graph of the function f(x) = 2x is a vector space.
The following example is important (it will appear frequently) and is our first example of what we could say is an “abstract vector space”. To emphasize, a vector space is a set that comes equipped with an operation of addition and scalar multiplication and these two operations satisfy the list of properties above.
Example 14.5. Let V = Pn[t] be the set of all polynomials in the variable t and of degree at most n:
Pn[t] = na0 + a1t + a2t2 + · · · + antn | a0, a1, . . . , an ∈ R,.
Is V a vector space?
Solution. Let u(t) = u0 + u1t + · · · + untn and let v(t) = v0 + v1t + · · · + vntn be polynomials in V. We define the addition of u and v as the new polynomial (u + v) as follows:
(u + v)(t) = u(t) + v(t) = (u0 + v0) + (u1 + v1)t + · · · + (un + vn)tn.
112
Vector Spaces
Then u + v is a polynomial of degree at most n and thus (u + v) ∈ Pn[t], and therefore this shows that Pn[t] is closed under addition. Now let α be a scalar, define a new polynomial (αu) as follows:
(αu)(t) = (αu0) + (αu1)t + · · · + (αun)tn
Then (αu) is a polynomial of degree at most n and thus (αu) ∈ Pn[t]; hence, Pn[t] is closed under scalar multiplication. The 0 vector in Pn[t] is the zero polynomial 0(t) = 0. One can verify that all other properties of the definition of a vector space also hold; for example, addition is commutative and associative, etc. Thus Pn[t] is a vector space.
Example 14.6. Let V = Mm×n be the set of all m×n matrices. Under the usual operations of addition of matrices and scalar multiplication, is Mn×m a vector space?
Solution. Given matrices A, B ∈ Mm×n and a scalar α, we defined the sum A+B by adding entry-by-entry, and αA by multiplying each entry of A by α. It is clear that the space Mm×n is closed under these two operations. The 0 vector in Mm×n is the matrix of size m × n having all entries equal to zero. It can be verified that all other properties of the definition of a vector space also hold. Thus, the set Mm×n is a vector space.
Example 14.7. The n-dimensional Euclidean space V = Rn under the usual operations of addition and scalar multiplication is vector space.
Example 14.8. Let V = C[a, b] denote the set of functions with domain [a, b] and co-domain
R that are continuous. Is V a vector space?
Frequently, one encounters a vector space W that is a subset of a larger vector space V. In this case, we would say that W is a subspace of V. Below is the formal definition.
Definition 14.9: Let V be a vector space. A subset W of V is called a subspace of V
if it satisfies the following properties:
αu is in W.
Example 14.10. Let W be the graph of the function f(x) = 2x:
W = {(x, y) ∈ R2 | y = 2x}.
Is W a subspace of V = R2?
113
Lecture 14
Solution. If x = 0 then y = 2 · 0 = 0 and therefore 0 = (0, 0) is in W. Let u = (a, 2a) and
v = (b, 2b) be elements of W. Then
` ˛¸ x
x
` ˛¸ x
x
u + v = (a, 2a) + (b, 2b) = (a + b, 2a + 2b) = (a + b, 2 (a + b)).
Because the x and y components of u + v satisfy y = 2x then u + v is inside in W. Thus, W
is closed under addition. Let α be any scalar and let u = (a, 2a) be an element of W. Then
`˛¸x
x
`˛¸x
x
αu = (αa, α2a) = ( αa , 2 (αa))
Because the x and y components of αu satisfy y = 2x then αu is an element of W, and thus W is closed under scalar multiplication. All three conditions of a subspace are satisfied for W and therefore W is a subspace of V.
Example 14.11. Let W be the first quadrant in R2:
W = {(x, y) ∈ R2 | x ≥ 0, y ≥ 0}.
Is W a subspace?
Solution. The set W contains the zero vector and the sum of two vectors in W is again in W; you may want to verify this explicitly as follows: if u1 = (x1, y1) is in W then x1 ≥ 0 and y1 ≥ 0, and similarly if u2 = (x2, y2) is in W then x2 ≥ 0 and y2 ≥ 0. Then the sum u1 +u2 = (x1 +x2, y1 +y2) has components x1 +y1 ≥ 0 and x2 +y2 ≥ 0 and therefore u1 +u2 is in W. However, W is not closed under scalar multiplication. For example if u = (1, 1) and α = −1 then αu = (−1, −1) is not in W because the components of αu are clearly not non-negative.
Example 14.12. Let V = Mn×n be the vector space of all n × n matrices. We define the
trace of a matrix A ∈ Mn×n as the sum of its diagonal entries:
tr(A) = a11 + a22 + · · · + ann.
Let W be the set of all n × n matrices whose trace is zero:
W = {A ∈ Mn×n | tr(A) = 0}.
Is W a subspace of V?
Solution. If 0 is the n × n zero matrix then clearly tr(0) = 0, and thus 0 ∈ Mn×n. Suppose that A and B are in W. Then necessarily tr(A) = 0 and tr(B) = 0. Consider the matrix C = A + B. Then
tr(C) = tr(A + B) = (a11 + b11) + (a22 + b22) + · · · + (ann + bnn)
= (a11 + · · · + ann) + (b11 + · · · + bnn)
= tr(A) + tr(B)
= 0
114
Vector Spaces
Therefore, tr(C) = 0 and consequently C = A + B ∈ W, in other words, W is closed under addition. Now let α be a scalar and let C = αA. Then
tr(C) = tr(αA) = (αa11) + (αa22) + · · · + (αann) = α tr(A) = 0.
Thus, tr(C) = 0, that is, C = αA ∈ W, and consequently W is closed under scalar multipli- cation. Therefore, the set W is a subspace of V.
Example 14.13. Let V = Pn[t] and consider the subset W of V:
W = {u ∈ Pn[t] | u′(1) = 0}
In other words, W consists of polynomials of degree n in the variable t whose derivative at
t = 1 is zero. Is W a subspace of V?
Solution. The zero polynomial 0(t) = 0 clearly has derivative at t = 1 equal to zero, that is, 0′(1) = 0, and thus the zero polynomial is in W. Now suppose that u(t) and v(t) are two polynomials in W. Then, u′(1) = 0 and also v′(1) = 0. To verify whether or not W is closed under addition, we must determine whether the sum polynomial (u + v)(t) has a derivative at t = 1 equal to zero. From the rules of differentiation, we compute
(u + v)′(1) = u′(1) + v′(1) = 0 + 0.
Therefore, the polynomial (u + v) is in W, and thus W is closed under addition. Now let α be any scalar and let u(t) be a polynomial in W. Then u′(1) = 0. To determine whether or not the scalar multiple αu(t) is in W we must determine if αu(t) has a derivative of zero at t = 1. Using the rules of differentiation, we compute that
(αu)′(1) = αu′(1) = α · 0 = 0.
Therefore, the polynomial (αu)(t) is in W and thus W is closed under scalar multiplication. All three properties of a subspace hold for W and therefore W is a subspace of Pn[t].
Example 14.14. Let V = Pn[t] and consider the subset W of V:
W = {u ∈ Pn[t] | u(2) = −1}
In other words, W consists of polynomials of degree n in the variable t whose value t = 2 is
−1. Is W a subspace of V?
Solution. The zero polynomial 0(t) = 0 clearly does not equal −1 at t = 2. Therefore, W does not contain the zero polynomial and, because all three conditions of a subspace must be satisfied for W to be a subspace, then W is not a subspace of Pn[t]. As an exercise, you may want to investigate whether or not W is closed under addition and scalar multiplication.
115
Lecture 14
Example 14.15. A square matrix A is said to be symmetric if AT = A. For example, here is a 3 × 3 symmetric matrix:
1
A = 2
4 5
2 −3
−3 5 7
Verify for yourself that we do indeed have that AT = A. Let W be the set of all symmetric
n × n matrices. Is W a subspace of V = Mn×n?
Example 14.16. For any vector space V, there are two trivial subspaces in V, namely, V itself is a subspace of V and the set consisting of the zero vector W = {0} is a subspace of V.
There is one particular way to generate a subspace of any given vector space V using the span of a set of vectors. Recall that we defined the span of a set of vectors in Rn but we can define the same notion on a general vector space V.
Definition 14.17: Let V be a vector space and let v1, v2, . . . , vp be vectors in V. The
span of {v1, . . . , vp} is the set of all linear combinations of v1, . . . , vp:
span{v1, v2, . . . , vp} = nt1v1 + t2v2 + · · · + vpvp | t1, t2, . . . , tp ∈ R,.
We now show that the span of a set of vectors in V is a subspace of V.
Theorem 14.18: If v1, v2, . . . , vp are vectors in V then span{v1, . . . , vp} is a subspace of
V.
Solution. Let u = t1v1+· · ·+tpvp and w = s1v1+· · ·+spvp be two vectors in span{v1, v2, . . . , vp}.
Then
u + w = (t1v1 + · · · + tpvp) + (s1v1 + · · · + spvp) = (t1 + s1)v1 + · · · + (tp + sp)vp.
Therefore u + w is also in the span of v1, . . . , vp. Now consider αu:
αu = α(t1v1 + · · · + tpvp) = (αt1)v1 + · · · + (αtp)vp.
Therefore, αu is in the span of v1, . . . , vp. Lastly, since 0v1 + 0v2 + · · · + 0vp = 0 then the
zero vector 0 is in the span of v1, v2, . . . , vp. Therefore, span{v1, v2, . . . , vp} is a subspace
of V.
Given a general subspace W of V, if w1, w2, . . . , wp are vectors in W such that
span{w1, w2, . . . , wp} = W
then we say that {w1, w2, . . . , wp} is a spanning set of W. Hence, every vector in W can be written as a linear combination of the vectors w1, w2, . . . , wp.
After this lecture you should know the following:
116
Vector Spaces
Lecture 15
Before we begin this Lecture, we review subspaces. Recall that W is a subspace of a vector space V if W is a subset of V and
In the previous lecture we gave several examples of subspaces. For example, we showed that a line through the origin in R2 is a subspace of R2 and we gave examples of subspaces of Pn[t] and Mn×m. We also showed that if v1, . . . , vp are vectors in a vector space V then
W = span{v1, v2, . . . , vp}
is a subspace of V.
In Lecture 7, we defined what it meant for a vector mapping T : Rn → Rm to be a linear mapping. We now want to introduce linear mappings on general vector spaces; you will notice that the definition is essentially the same but the key point to remember is that the underlying spaces are not Rn but a general vector space.
Definition 15.1: Let T : V → U be a mapping of vector spaces. Then T is called a linear mapping if
Example 15.2. Let V = Mn×n be the vector space of n × n matrices and let T : V → V be the mapping
T(A) = A + AT .
117
Lecture 15
Linear Maps
118
Linear Maps
Is T is a linear mapping?
Solution. Let A and B be matrices in V. Then using the properties of the transpose and regrouping we obtain:
T(A + B) = (A + B) + (A + B)T
= A + B + AT + BT
= (A + AT ) + (B + BT )
= T(A) + T(B).
Similarly, if α is any scalar then
T(αA) = (αA) + (αA)T
= αA + αAT
= α(A + AT )
= αT(A).
This proves that T satisfies both conditions of Definition 15.1 and thus T is a linear mapping.
Example 15.3. Let V = Mn×n be the vector space of n × n matrices, where n ≥ 2, and let
T(A) = det(A)
T : V → R be the mapping
Is T is a linear mapping?
Solution. If T is a linear mapping then according to Definition 15.1, we must have T(A + B) = det(A + B) = det(A) + det(B) and also T(αA) = αT(A) for any scalar α. Do these properties actually hold though? For example, we know from the properties of the determinant that det(αA) = αn det(A) and therefore it does not hold that T(αA) = αT(A) unless α = 1. Therefore, T is not a linear mapping. Also, it does not hold in general that det(A + B) = det(A) + det(B); in fact it rarely holds. For example, if
A =
, B =
−1 1
2 0
0 1 0 3
then det(A) = 2, det(B) = −3 and therefore det(A) + det(B) = −1. On the other hand,
A + B =
1 1
0 4
and thus det(A + B) = 4. Thus, det(A + B) /= det(A) + det(B).
Example 15.4. Let V = Pn[t] be the vector space of polynomials in the variable t of degree no more than n ≥ 1. Consider the mapping T : V → V define as
T(f(t)) = 2f(t) + f′(t).
119
Lecture 15
For example, if f(t) = 3t6 − t2 + 5 then
T(f(t)) = 2f(t) + f′(t)
= 2(3t5 − t2 + 5) + (18t5 − 2t)
= 6t5 + 18t5 − 2t2 − 2t + 10.
Is T is a linear mapping?
Solution. Let f(t) and g(t) be polynomials of degree no more than n ≥ 1. Then
T(f(t) + g(t)) = 2(f(t) + g(t)) + (f(t) + g(t))′
= 2f(t) + 2g(t) + f′(t) + g′(t)
= (2f(t) + f′(t)) + (2g(t) + g′(t))
= T(f(t)) + T(g(t)).
Therefore, T(f(t) + g(t)) = T(f(t)) + T(g(t)). Now let α be any scalar. Then
T(αf(t)) = 2(αf(t)) + (αf(t))′
= 2αf(t) + αf′(t)
= α(2f(t) + f′(t))
= αT(f(t)).
Therefore, T(αf(t)) = αT(f(t)). Therefore, T is a linear mapping.
We now introduce two important subsets associated to a linear mapping.
Definition 15.5: Let T : V → U be a linear mapping.
ker(T) = {v ∈ V | T(v) = 0}.
Range(T) = {b ∈ U | there exists some v ∈ U such that T(v) = b}.
You may have noticed that the definition of the range of a linear mapping on an abstract vector space is the usual definition of the range of a function. Not surprisingly, the kernel and range are subspaces of the domain and codomain, respectively.
120
Linear Maps
Theorem 15.6: Let T : V → U be a linear mapping. Then ker(T) is a subspace of V and Range(T) is a subspace of U.
Proof. Suppose that v and u are in ker(T). Then T(v) = 0 and T(u) = 0. Then by linearity of T it holds that
T(v + u) = T(v) + T(u) = 0 + 0 = 0.
Therefore, since T(u + v) = 0 then u + v is in ker(T). This shows that ker(T) is closed under addition. Now suppose that α is any scalar and v is in ker(T). Then T(v) = 0 and thus by linearity of T it holds that
T(αv) = αT(v) = α0 = 0.
Therefore, since T(αv) = 0 then αv is in ker(T) and this proves that ker(T) is closed under scalar multiplication. Lastly, by linearity of T it holds that
T(0) = T(v − v) = T(v) − T(v) = 0
that is, T(0) = 0. Therefore, the zero vector 0 is in ker(T). This proves that ker(T) is a subspace of V. The proof that Range(T) is a subspace of U is left as an exercise.
Example 15.7. Let V = Mn×n be the vector space of n × n matrices and let T : V → V be the mapping
T(A) = A + AT .
Describe the kernel of T.
Solution. A matrix A is in the kernel of T if T(A) = A + AT = 0, that is, if AT = −A. Hence,
ker(A) = {A ∈ Mn×n | AT = −A}.
What type of matrix A satisfies AT = −A? For example, consider the case that A is the 2 × 2 matrix
A =
a11 a12
a21 a22
and AT = −A. Then
a11 a21
a12 a22
=
−a11 −a12
−a21 −a22
.
Therefore, it must hold that a11 = −a11, a21 = −a12 and a22 = −a22. Then necessarily
a11 = 0 and a22 = 0 and a12 can be arbitrary. For example, the matrix
A =
0 7
−7 0
satisfies AT = −A. Using a similar computation as above, a 3 × 3 matrix satisfies AT = −A
if A is of the form
a
A = −a 0 c
−b −c 0
0 b
121
Lecture 15
where a, b, c are arbitrary constants. In general, a matrix A that satisfies AT = −A is called
skew-symmetric.
Example 15.8. Let V be the vector space of differentiable functions on the interval [a, b]. That is, f is an element of V if f : [a, b] → R is differentiable. Describe the kernel of the linear mapping T : V → V defined as
T(f(x)) = f(x) + f′(x).
Solution. A function f is in the kernel of T if T(f(x)) = 0, that is, if f(x) + f′(x) = 0. Equivalently, if f′(x) = −f(x). What functions f do you know of satisfy f′(x) = −f(x)? How about f(x) = e−x? It is clear that f′(x) = −e−x = −f(x) and thus f(x) = e−x is in ker(T). How about g(x) = 2e−x? We compute that g′(x) = −2e−x = −g(x) and thus g is also in ker(T). It turns out that the elements of ker(T) are of the form f(x) = Ce−x for a constant C.
15.2 Null space and Column space
In the previous section, we introduced the kernel and range of a general linear mapping T : V → U. In this section, we consider the particular case of matrix mappings TA : Rn → Rm for some m×n matrix A. In this case, v is in the kernel of TA if and only if TA(v) = Av = 0. In other words, v ∈ ker(TA) if and only if v is a solution to the homogeneous system Ax = 0. Because the case when T is a matrix mapping arises so frequently, we give a name to the set of vectors v such that Av = 0.
Definition 15.9: The null space of a matrix A ∈ Mm×n, denoted by Null(A), is the subset of Rn consisting of vectors v such that Av = 0. In other words, v ∈ Null(A) if and only if Av = 0. Using set notation:
Null(A) = {v ∈ Rn | Av = 0}.
Hence, the following holds
ker(TA) = Null(A).
Because the kernel of a linear mapping is a subspace we obtain the following.
Theorem 15.10: If A ∈ Mm×n then Null(A) is a subspace of Rn.
Hence, by Theorem 15.10, if u and v are two solutions to the linear system Ax = 0 then
αu + βv is also a solution:
A(αu + βv) = αAu + βAv = α · 0 + β · 0 = 0.
122
Linear Maps
Example 15.11. Let V = R4 and consider the following subset of V:
W = {(x1, x2, x3, x4) ∈ R4 | 2x1 − 3x2 + x3 − 7x4 = 0}.
Is W a subspace of V?
Solution. The set W is the null space of the matrix 1 × 4 matrix A given by
A = 2 −3 1 −7 .
Hence, W = Null(A) and consequently W is a subspace.
From our previous remarks, the null space of a matrix A ∈ Mm×n is just the solution set of the homogeneous system Ax = 0. Therefore, one way to explicitly describe the null space of A is to solve the system Ax = 0 and write the general solution in parametric vector form. From our previous work on solving linear systems, if the rref(A) has r leading 1’s then the number of parameters in the solution set is d = n − r. Therefore, after performing back substitution, we will obtain vectors v1, . . . , vd such that the general solution in parametric vector form can be written as
x = t1v1 + t2v2 + · · · + tdvd
where t1, t2, . . . , td are arbitrary numbers. Therefore,
Null(A) = span{v1, v2, . . . , vd}.
Hence, the vectors v1, v2, . . . , vn form a spanning set for Null(A).
Example 15.12. Find a spanning set for the null space of the matrix
−3 6 −1 1 −7
A = 1 −2 2 3 −1 .
2 −4 5 8 −4
Solution. The null space of A is the solution set of the homogeneous system Ax = 0. Performing elementary row operations one obtains
−2 0 −1
1 3
A ~ 0 0 1 2 −2 .
0 0 0 0 0
Clearly r = rank(A) and since n = 5 we will have d = 3 vectors in a spanning set for Null(A). Letting x5 = t1, and x4 = t2, then from the 2nd row we obtain
x3 = −2t2 + 2t1.
Letting x2 = t3, then from the 1st row we obtain
x1 = 2t3 + t2 − 3t1.
123
Lecture 15
1
0
1
1
0
2 3
x = t 2 + t −2 + t 0
0
0
Writing the general solution in parametric vector form we obtain
−3 1 2
0 0 1
Therefore,
Null(A) = span
0
1
v
1
2 , −2 0
1 0
` ˛¸ x ` ˛¸ x `˛¸x
v v
2 3
−3 1 2
0 0 1
0 0
You can verify that Av1 = Av2 = Av3 = 0.
Now we consider the range of a matrix mapping TA : Rn → Rm. Recall that a vector b in the co-domain Rm is in the range of TA if there exists some vector x in the domain Rn such that TA(x) = b. Since, TA(x) = Ax then Ax = b. Now, if A has columns A = v1 v2 · · · vn and x = (x1, x2, . . . , xn) then recall that
Ax = x1v1 + x2v2 + · · · + xnvn
and thus Ax = x1v1 + x2v2 + · · · + xnvn = b. Thus, a vector b is in the range of A if it can be written as a linear combination of the columns v1, v2, . . . , vn of A. This motivates the following definition.
Definition 15.13: Let A ∈ Mm×n be a matrix. The span of the columns of A is called the column space of A. The column space of A is denoted by Col(A). Explicitly, if A = v1 v2 · · · vn then
Col(A) = span{v1, v2, . . . , vn}.
In summary, we can write that
Range(TA) = Col(A). and since Range(TA) is a subspace of Rm then so is Col(A).
Theorem 15.14: The column space of a m × n matrix is a subspace of Rm.
124
Linear Maps
Example 15.15. Let
2
4 −2
A = −
1
2
3 7 −8 6
−5 7 3 ,
b = −1 .
3
3
Is b in the column space Col(A)?
Solution. The vector b is in the column space of A if there exists x ∈ R4 such that Ax = b. Hence, we must determine if Ax = b has a solution. Performing elementary row operations on the augmented matrix A b we obtain
2 4 −2 1
A b ~ 0 1 −5 −4 −2
3
0 0 0 17 1
The system is consistent and therefore Ax = b will have a solution. Therefore, b is in Col(A).
After this lecture you should know the following:
Lecture 16
Linear Independence, Bases, and Dimension
16.1 Linear Independence
Roughly speaking, the concept of linear independence evolves around the idea of working with “efficient” spanning sets for a subspace. For instance, the set of directions
{EAST, NORTH, NORTH-EAST}
are redundant since a total displacement in the NORTH-EAST direction can be obtained by combining individual NORTH and EAST displacements. With these vague statements out of the way, we introduce the formal definition of what it means for a set of vectors to be “efficient”.
Definition 16.1: Let V be a vector space and let {v1, v2, . . . , vp} be a set of vectors in
V. Then {v1, v2, . . . , vp} is linearly independent if the only scalars c1, c2, . . . , cp that satisfy the equation
c1v1 + c2v2 + · · · + cpvp = 0
are the trivial scalars c1 = c2 = · · · = cp = 0. If the set {v1, . . . , vp} is not linearly independent then we say that it is linearly dependent.
We now describe the redundancy in a set of linear dependent vectors. If {v1, . . . , vp} are linearly dependent, it follows that there are scalars c1, c2, . . . , cp, at least one of which is nonzero, such that
c1v1 + c2v2 + · · · + cpvp = 0. (⋆)
For example, suppose that {v1, v2, v3, v4} are linearly dependent. Then there are scalars c1, c2, c3, c4, not all of them zero, such that equation (⋆) holds. Suppose, for the sake of argument, that c3 /= 0. Then,
c
3
v = − v
3 1
c1 c2
c
3
2
c4
c
3
4
— v − v .
126
Linear Independence, Bases, and Dimension
Therefore, when a set of vectors is linearly dependent, it is possible to write one of the vec- tors as a linear combination of the others. It is in this sense that a set of linearly dependent vectors are redundant. In fact, if a set of vectors are linearly dependent we can say even more as the following theorem states.
Theorem 16.2: A set of vectors {v1, v2, . . . , vp}, with v1 /= 0, is linearly dependent if and only if some vj is a linear combination of the preceding vectors v1, . . . , vj−1.
Example 16.3. Show that the following set of 2 × 2 matrices is linearly dependent:
1
A =
, A =
2 3
1 2 −1 3 5 0
0 −1 1 0 −2 −3
, A = .
Solution. It is clear that A1 and A2 are linearly independent, i.e., A1 cannot be written as a scalar multiple of A2, and vice-versa. Since the (2, 1) entry of A1 is zero, the only way to get the −2 in the (2, 1) entry of A3 is to multiply A2 by −2. Similary, since the (2, 2) entry of A2 is zero, the only way to get the −3 in the (2, 2) entry of A3 is to multiply A1 by 3. Hence, we suspect that 3A1 − 2A2 = A3. Verify:
3A1 − 2A2 =
—
0 −3 2
5 0
3 6 −2 6
0 −2 −3
= = A
3
Therefore, 3A1 − 2A2 − A3 = 0 and thus we have found scalars c1, c2, c3 not all zero such that c1A1 + c2A2 + c3A3 = 0.
16.2 Bases
We now introduce the important concept of a basis. Given a set of vectors {v1, . . . , vp−1, vp} in V, we showed that W = span{v1, v2, . . . , vp} is a subspace of V. If say vp is linearly dependent on v1, v2, . . . , vp−1 then we can remove vp and the smaller set {v1, . . . , vp−1} still spans all of W:
W = span{v1, v2, . . . , vp−1, vp} = span{v1, . . . , vp−1}.
Intuitively, vp does not provide an independent “direction” in generating W. If some other vector vj is linearly dependent on v1, . . . , vp−1 then we can remove vj and the resulting smaller set of vectors still spans W. We can continue removing vectors until we obtain a minimal set of vectors that are linearly independent and still span W. The following remarks motivate the following important definition.
Definition 16.4: Let W be a subspace of a vector space V. A set of vectors B = {v1, . . . , vk}
in W is said to be a basis for W if
(a) the set B spans all of W, that is, W = span{v1, . . . , vk}, and
127
Lecture 16
(b) the set B is linearly independent.
A basis is therefore a minimal spanning set for a subspace. Indeed, if B = {v1, . . . , vp} is a basis for W and we remove say vp, then B˜ = {v1, . . . , vp−1} cannot be a basis for W. Why? If B = {v1, . . . , vp} is a basis then it is linearly independent and therefore vp cannot be written as a linear combination of the others. In other words, vp ∈ W is not in the span of B˜ = {v1, . . . , vp−1} and therefore B˜ is not a basis for W because a basis must be a spanning set. If, on the other hand, we start with a basis B = {v1, . . . , vp} for W and we add a new vector u from W then B˜ = {v1, . . . , vp, u} is not a basis for W. Why? We still have that span B˜ = W but now B˜ is not linearly independent. Indeed, because B = {v1, . . . , vp} is a basis for W, the vector u can be written as a linear combination of {v1, . . . , vp}, and thus B˜ is not linearly independent.
Example 16.5. Show that the standard unit vectors form a basis for V = R3:
1 0 0
1
e = 0 ,
2 3
e = 1 , e = 0
0 0 1
Solution. Any vector x ∈ R3 can be written as a linear combination of e1, e2, e3:
x1 1 0 0
x = x = x 0 + x 1 + x 0 = x e + x e + x e
2 1 2 3 1 1 2 2 3 3
x3 0 0 1
Therefore, span{e1, e2, e3} = R3. The set B = {e1, e2, e3} is linearly independent. Indeed, if there are scalars c1, c2, c3 such that
c1e1 + c2e2 + c3e3 = 0
then clearly they must all be zero, c1 = c2 = c3 = 0. Therefore, by definition, B = {e1, e2, e3} is a basis for R3. This basis is called the standard basis for R3. Analogous arguments hold for {e1, e2, . . . , en} in Rn.
Example 16.6. Is B = {v1, v2, v3} a basis for R3?
2 −4 4
1 2
v = 0 , v = −
−4 8
3
2 , v = −6
−6
Solution. Form the matrix A = [v1 v2 v3] and row reduce:
A ~ 0
1 0 0
1 0
0 0 1
128
Linear Independence, Bases, and Dimension
Therefore, the only solution to Ax = 0 is the trivial solution. Therefore, B is linearly inde-
pendent. Moreover, for any b ∈ R3, the augmented matrix A b is consistent. Therefore,
the columns of A span all of R3:
Col(A) = span{v1, v2, v3} = R3.
Therefore, B is a basis for R3.
Example 16.7. In V = R4, consider the vectors
1
v =
1 2
0
2
, v =
3 −1
−2
3
, v =
4
2
−1
.
−2 1 −3
Let W = span{v1, v2, v3}. Is B = {v1, v2, v3} a basis for W?
Solution. By definition, B is a spanning set for W, so we need only determine if B is linearly independent. Form the matrix, A = [v1 v2 v3] and row reduce to obtain
A ~
1
0
0
| 0 | 1 |
1 | −1 | |
0 | 0 | |
0 | 0 | 0 |
Hence, rank(A) = 2 and thus B is linearly dependent. Notice v1 − v2 = v3. Therefore, B is not a basis of W.
Example 16.8. Find a basis for the vector space of 2 × 2 matrices.
Example 16.9. Recall that a n × n is skew-symmetric A if AT = −A. We proved that the set of n × n matrices is a subspace. Find a basis for the set of 3 × 3 skew-symmetric matrices.
16.3 Dimension of a Vector Space
The following theorem will lead to the definition of the dimension of a vector space.
Theorem 16.10: Let V be a vector space. Then all bases of V have the same number of vectors.
Proof: We will prove the theorem for the case that V = Rn. We already know that the standard unit vectors {e1, e2, . . . , en} is a basis of Rn. Let {u1, u2, . . . , up} be nonzero vec- tors in Rn and suppose first that p > n. In Lecture 6, Theorem 6.7, we proved that any set of vectors in Rn containing more than n vectors is automatically linearly dependent. The reason is that the RREF of A = u1 u2 · · · up will contain at most r = n leading ones,
129
Lecture 16
and therefore d = p − n > 0. Therefore, the solution set of Ax = 0 contains non-trivial solutions. On the other hand, suppose instead that p < n. In Lecture 4, Theorem 4.11, we proved that a set of vectors {u1, . . . , up} in Rn spans Rn if and only if the RREF of A has exactly r = n leading ones. The largest possible value of r is r = p < n. Therefore, if p < n then {u1, u2, . . . , up} cannot be a basis for Rn. Thus, in either case (p > n or p < n), the set
{u1, u2, . . . , up} cannot be a basis for Rn. Hence, any basis in Rn must contain n vectors.
The previous theorem does not say that every set {v1, v2, . . . , vn} of nonzero vectors in
Rn containing n vectors is automatically a basis for Rn. For example,
1 0 2
1
0
v = 0 ,
2
0
3
v = 1 , v = 3
0
do not form a basis for R3 because
x = 0
0
1
is not in the span of {v1, v2, v3}. All that we can say is that a set of vectors in Rn containing fewer or more than n vectors is automatically not a basis for Rn. From Theorem 16.10, any basis in Rn must have exactly n vectors. In fact, on a general abstract vector space V, if
{v1, v2, . . . , vn} is a basis for V then any other basis for V must have exactly n vectors also. Because of this result, we can make the following definition.
Definition 16.11: Let V be a vector space. The dimension of V, denoted dim V, is the number of vectors in any basis of V. The dimension of the trivial vector space V = {0} is defined to be zero.
There is one subtle issue we are sweeping under the rug: Does every vector space have a basis? The answer is yes but we will not prove this result here.
Moving on, suppose that we have a set B = {v1, v2, . . . , vn} in Rn containing exactly n vectors. For B = {v1, v2, . . . , vn} to be a basis of Rn, the set B must be linearly independent and span B = Rn. In fact, it can be shown that if B is linearly independent then the spanning condition span B = Rn is automatically satisfied, and vice-versa. For example, say the vec- tors {v1, v2, . . . , vn} in Rn are linearly independent, and put A = [v1 v2 · · · vn]. Then A−1 exists and therefore Ax = b is always solvable. Hence, Col(A) = span {v1, v2, . . . , vn} = Rn. In summary, we have the following theorem.
Theorem 16.12: Let B = {v1, . . . , vn} be vectors in Rn. If B is linearly independent then B is a basis for Rn. Or if span{v1, v2, . . . , vn} = Rn then B is a basis for Rn.
130
Linear Independence, Bases, and Dimension
Example 16.13. Do the columns of the matrix A form a basis for R4?
A =
2 3 3 −2
4 | 7 | 8 | −6 |
0 | 0 | 1 | 0 |
−4 | −6 | −6 | 3 |
A =
Solution. Let v1, v2, v3, v4 denote the columns of A. Since we have n = 4 vectors in Rn, we need only check that they are linearly independent. Compute
det A = −2 /= 0
Hence, rank(A) = 4 and thus the columns of A are linearly independent. Therefore, the vectors v1, v2, v3, v4 form a basis for R4.
A subspace W of a vector space V is a vector space in its own right, and therefore also has dimension. By definition, if B = {v1, . . . , vk} is a linearly independent set in W and span{v1, . . . , vk} = W, then B is a basis for W and in this case the dimension of W is k. Since an n-dimensional vector space V requires exactly n vectors in any basis, then if W is a strict subspace of V then
dim W < dim V.
As an example, in V = R3 subspaces can be classified by dimension:
Example 16.14. Find a basis for Null(A) and the dim Null(A) if
−2 4 −2 −4
−3 8 2 −3
2 −6 −3 1 .
Solution. By definition, the Null(A) is the solution set of the homogeneous system Ax = 0. Row reducing we obtain
A ~ 0
1 0 6 5
1 5/2 3/2
0 0 0 0
131
Lecture 16
x = t
−3/2
0
1
+ s
−5/2
1
0
The general solution to Ax = 0 in parametric form is
−5 −6
= tv1 + sv2
By construction, the vectors
1
v =
−3/2
0
1
−5
2
, v =
−5/2
1
0
−6
span the null space (A) and they are linearly independent. Therefore, B = {v1, v2} is a basis for Null(A) and therefore dim Null(A) = 2. In general, the dimension of the Null(A) is the number of free parameters in the solution set of the system Ax = 0, that is,
dim Null(A) = d = n − rank(A)
Example 16.15. Find a basis for Col(A) and the dim Col(A) if
A =
1 2 3 −4 8
1 2 0 2 8
2 4 −3 10 9
3 6 0 6 9
.
Solution. By definition, the column space of A is the span of the columns of A, which we denote by A = [v1 v2 v3 v4 v5]. Thus, to find a basis for Col(A), by trial and error we could determine the largest subset of the columns of A that are linearly independent. For example, first we determine if {v1, v2} is linearly independent. If yes, then add v3 and determine if
{v1, v2, v3} is linearly independent. If {v1, v2} is not linearly independent then discard v2
and determine if {v1, v3} is linearly independent. We continue this process until we have
determined the largest subset of the columns of A that is linearly independent, and this will yield a basis for Col(A). Instead, we can use the fact that matrices that are row equivalent induce the same solution set for the associated homogeneous system. Hence, let B be the RREF of A:
B = rref(A) =
1 2 0 2 0
0 0 1 −2 0
0 0 0 0 1
0 0 0 0 0
132
Linear Independence, Bases, and Dimension
By inspection, the columns b1, b3, b5 of B are linearly independent. It is easy to see that
b2 = 2b1 and b4 = 2b1 − 2b3. These same linear relations hold for the columns of A:
A =
1 2 3 −4 8
1 2 0 2 8
2 4 −3 10 9
3 6 0 6 9
By inspection, v2 = 2v1 and v4 = 2v1 − 2v3. Thus, because b1, b3, b5 are linearly inde- pendent columns of B =rref(A), then v1, v3, v5 are linearly independent columns of A. Therefore, we have
1 3 8
1 3 5
Col(A) = span{v , v , v } = span
2
,
−3
9
1 0 , 8
3 0 9
and consequently dim Col(A) = 3. This procedure works in general: To find a basis for the Col(A), row reduce A ~ B until you can determine which columns of B are linearly independent. The columns of A in the same position as the linearly independent columns of B form a basis for the Col(A).
WARNING: Do not take the linearly independent columns of B as a basis for Col(A). Always go back to the original matrix A to select the columns.
After this lecture you should know the following:
Lecture 17
17.1 The Rank of a Matrix
We now give the definition to the rank of a matrix.
Definition 17.1: The rank of a matrix A is the dimension of its column space. We will use rank(A) to denote the rank of A.
Recall that Col(A) = Range(TA), and thus the rank of A is the dimension of the range of the linear mapping TA. The range of a mapping is sometimes called the image.
We now define the nullity of a matrix.
Definition 17.2: The nullity of a matrix A is the dimension of its nullspace Null(A). We will use nullity(A) to denote the nullity of A.
Recall that (A) = ker(TA), and thus the nullity of A is the dimension of the kernel of the linear mapping TA.
The rank and nullity of a matrix are connected via the following fundamental theorem known as the Rank Theorem.
Theorem 17.3: (Rank Theorem) Let A be a m × n matrix. The rank of A is the number of leading 1’s in its RREF. Moreover, the following equation holds:
n = rank(A) + nullity(A).
Proof. A basis for the column space is obtained by computing rref(A) and identifying the columns that contain a leading 1. Each column of A corresponding to a column of rref(A) with a leading 1 is a basis vector for the column space of A. Therefore, if r is the number of leading 1’s then r = rank(A). Now let d = n − r. The number of free parameters in the
The Rank Theorem
134
The Rank Theorem
solution set of Ax = 0 is d and therefore a basis for Null(A) will contain d vectors, that is, nullity(A) = d. Therefore,
nullity(A) = n − rank(A).
Example 17.4. Find the rank and nullity of the matrix
1 −2 2 3 −6
A = 0 −1 −3 1 1 .
−2 4 −3 −6 11
Solution. Row reduce far enough to identify where the leading entries are:
2R1 +R2
1
A −−−−→ 0
−1 −3 1 1
−2 2 3 −6
0 0 1 0 −1
There are r = 3 leading entries and therefore rank(A) = 3. The nullity is therefore nullity(A) = 5 − rank(A) = 2.
Example 17.5. Find the rank and nullity of the matrix
A = −1 4 2
1 −3 −1
.
−1 3 0
Solution. Row reduce far enough to identify where the leading entries are:
R1+R2 ,R1+R3
A −−−−−−−−→ 0 1 1
1
−3 −1
0 0 −1
There are r = 3 leading entries and therefore rank(A) = 3. The nullity is therefore nullity(A) = 3 − rank(A) = 0. Another way to see that nullity(A) = 0 is as follows. From the above computation, A is invertible. Therefore, there is only one vector in Null(A) = {0}. The subspace {0} has dimension zero.
Using the rank and nullity of a matrix, we now provide further characterizations of invertible matrices.
Theorem 17.6: Let A be a n × n matrix. The following statements are equivalent:
135
Lecture 17
After this lecture you should know the following:
136
The Rank Theorem
Lecture 18
Coordinate Systems
Recall that a basis of a vector space V is a set of vectors B = {v1, v2, . . . , vn} in V such that
Hence, if B is a basis for V, each vector x∗ ∈ V can be written as a linear combination of B:
x∗ = c1v1 + c2v2 + · · · + cnvn.
Moreover, from the definition of linear independence given in Definition 6.1, any vector x ∈ span(B) can be written in only one way as a linear combination of v1, . . . , vn. In other words, for the x∗ above, there does not exist other scalars t1, . . . , tn such that also
x∗ = t1v1 + t2v2 + · · · + tnvn.
To see this, suppose that we can write x∗ in two different ways using B:
x∗ = c1v1 + c2v2 + · · · + cnvn
x∗ = t1v1 + t2v2 + · · · + tnvn.
Then
0 = x∗ − x∗ = (c1 − t1)v1 + (c2 − t2)v2 + · · · + (cn − tn)vn.
Since B = {v1, . . . , vn} is linearly independent, the only linear combination of v1, . . . , vn that gives the zero vector 0 is the trivial linear combination. Therefore, it must be the case that ci − ti = 0, or equivalently that ci = ti for all i = 1, 2 . . . , n. Thus, there is only one way to write x∗ in terms of B = {v1, . . . , vn}. Hence, relative to the basis B = {v1, v2, . . . , vn}, the scalars c1, c2, . . . , cn uniquely determine the vector x, and vice-versa.
Our preceding discussion on the unique representation property of vectors in a given basis leads to the following definition.
138
B
[x] =
c
2
.
.
Coordinate Systems
Definition 18.1: Let B = {v1, . . . , vn} be a basis for V and let x ∈ V. The coordinates of x relative to the basis B are the unique scalars c1, c2, . . . , cn such that
x = c1v1 + c2v2 + · · · + cnvn.
In vector notation, the B-coordinates of x will be denoted by
c1
cn
and we will call [x]B the coordinate vector of x relative to B.
The notation [x]B indicates that these are coordinates of x with respect to the basis B. If it is clear what basis we are working with, we will omit the subscript B and simply write
[x] for the coordinates of x relative to B.
Example 18.2. One can verify that
B =
1
,
1 −1
1
2
is a basis for R . Find the coordinates of v =
3
1
relative to B.
Solution. Let v1 = (1, 1) and let v2 = (−1, 1). By definition, the coordinates of v with respect to B are the scalars c1, c2 such that
1 1 2 2
v = c v + c v =
1 −1 c
1
1 1 c2
If we put P = [v1 v2], and let [v]B = (c1, c2), then we need to solve the linear system
v = P[v]B
Solving the linear system, one finds that the solution is [v]B = (2, −1), and therefore this is the B-coordinate vector of v, or the coordinates of v, relative to B.
It is clear how the procedure of the previous example can be generalized. Let B =
{v1, v2, . . . , vn} be a basis for Rn and let v be any vector in Rn. Put P = v1 v2 · · ·
vn .
Then the B-coordinates of v is the unique column vector [v]B solving the linear system
Px = v
139
Lecture 18
that is, x = [v]B is the unique solution to Px = v. Because v1, v2, . . . , vn are linearly independent, the solution to Px = v is
[v]B = P−1v.
We remark that if an inconsistent row arises when you row reduce the augmented matrix
P v then you have made an error in your row reduction algorithm. In summary, to find
coordinates with respect to a basis B in Rn, we need to solve a square linear system.
Example 18.3. Let
3 −1 3
1
v =
2
6 , v 0
= , x = 12
2 1 7
and let B = {v1, v2}. One can show that B is linearly independent and therefore a basis for
W = span{v1, v2}. Determine if x is in W, and if so, find the coordinate vector of x relative to B.
Solution. By definition, x is in W = span{v1, v2} if we can write x as a linear combination
of v1, v2:
x = c1v1 + c2v2
Form the associated augmented matrix and row reduce:
3 3 1
−1 0
6 0 12 ~ 0
2
1 3
2 1 7 0 0 0
The system is consistent with solution c1 = 2 and c2 = 3.
Therefore, x is in W, and the
B-coordinates of x are
[x]B =
2
3
Example 18.4. What are the coordinates of
3
v = 11
−7
in the standard basis E = {e1, e2, e3}?
Solution. Clearly,
v =
3 1
−7
0
11 = 3 0 + 11
0 0
0
1 − 7 0
1
140
Coordinate Systems
Therefore, the coordinate vector of v relative to {e1, e2, e3} is
3
E
[v] = 11
−7
Example 18.5. Let P3[t] be the vector space of polynomials of degree at most 3.
2 3
2 3
Solution. The set B = {1, t, t2, t3} is a spanning set for P3[t]. Indeed, any polynomial u(t) = c0 + c1t + c2t2 + c3t3 is clearly a linear combination of 1, t, t2, t3. Is B linearly independent? Suppose that there exists scalars c0, c1, c2, c3 such that
c0 + c1t + c2t2 + c3t3 = 0.
Since the above equality must hold for all values of t, we conclude that c0 = c1 = c2 = c3 = 0. Therefore, B is linearly independent, and consequently a basis for P3[t]. In the basis B, the coordinates of v(t) = 3 − t2 − 7t3 are
3
B
[v(t)] =
0
−1
−7
The basis B = {1, t, t2, t3} is called the standard basis in P3[t].
Example 18.6. Show that
B =
, ,
,
1 0 0 1 0 0 0 0
0 0 0 0 1 0 0 1
is a basis for M2×2. Find the coordinates of A =
3 0
−4 −1
relative to B.
Solution. Any matrix M =
11
m m
12
m21 m22
can be written as a linear combination of the ma-
trices in B:
11
m m
12
m21 m22
= m
11
1 0
0 0
+ m
12
0 1
0 0
+ m
21
0 0
1 0
+ m
22
0 0
0 1
If
c
1
1 0
0 0
+ c
2
0 1
0 0
+ c
3
0 0
1 0
+ c
4
0 0
0 1
=
c c
1 2
c3 c4
=
0 0
0 0
141
Lecture 18
1 2 3 4
then clearly c = c = c = c = 0. Therefore, B is linearly independent, and consequently
a basis for M2×2. The coordinates of A =
3 0
−4 −1
in the basis
B =
,
, ,
1 0 0 1 0 0 0 0
0 0 0 0 1 0 0 1
are
B
[A] =
3
0
−4
−1
The basis B above is the standard basis of M2×2.
18.2 Coordinate Mappings
Let B = {v1, v2, . . . , vn} be a basis of Rn and let P = [v1 v2 · · · vn] ∈ Mn×n. If x ∈ Rn and [x]B are the B-coordinates of x relative to B then
x = P[x]B. (⋆)
Hence, thinking of P : Rn → Rn as a linear mapping, P maps B-coordinate vectors to
coordinate vectors relative to the standard basis of Rn. For this reason, we call P the change-of-coordinates matrix from the basis B to the standard basis in Rn. If we need to emphasize that P is constructed from the basis B we will write PB instead of just P. Multiplying equation (⋆) by P−1 we obtain
P−1x = [x]B.
Therefore, P−1 maps coordinate vectors in the standard basis to coordinates relative to B.
Example 18.7. The columns of the matrix P form a basis B for R3:
1 3
P = −
3
1 −4 −2 .
0 0 −1
Solution. The matrix P maps B-coordinates to standard coordinates in R3. Therefore,
−2
B
x = P[x] =
1
1
142
Coordinate Systems
On the other hand, the inverse matrix P−1 maps standard coordinates in R3 to B-coordinates.
One can verify that
−1
4 3
P = −
1 −1 −1
6
0 0 −1
Therefore, the B coordinates of v are
−1
4 3
[v]B = P v = −
6 2 5
1 −1 −1 −
0 0 −1 0
1 = −1
0
When V is an abstract vector space, e.g. Pn[t] or Mn×n, the notion of a coordinate mapping is similar as the case when V = Rn. If V is an n-dimensional vector space and B = {v1, v2, . . . , vn} is a basis for V, we define the coordinate mapping P : V → Rn relative to B as the mapping
P(v) = [v]B.
Example 18.8. Let V = M2×2 and let B = {A1, A2, A3, A4} be the standard basis for
M2×2. What is P : M2×2 → R4?
Solution. Recall,
1 2 3 4
B = {A , A , A , A } =
,
,
1 0
,
0 0
0
1 0 0
0 0 0 0 1 0 0 1
Then for any A =
11
a a
12
a21 a22
we have
P
a11 a12
a21 a22
=
a
12
a
21
a22
a11
.
18.3 Matrix Representation of a Linear Map
Let V and W be vector spaces and let T : V → W be a linear mapping. Then by definition of a linear mapping, T(v + u) = T(v) + T(u) and T(αv) = αT(v) for every v, u ∈ V and α ∈ R. Let B = {v1, v2, . . . , vn} be a basis of V and let γ = {w1, w2, . . . , wm} be a basis of
W. Then for any v ∈ V there exists scalars c1, c2, . . . , cn such that
v = c1v1 + c2v2 + · · · + cnvn
143
Lecture 18
and thus [v]B = (c1, c2, . . . , cn) are the coordinates of v in the basis B By linearity of the mapping T we have
T(v) = T(c1v1 + c2v2 + · · · + cnvn)
= c1T(v1) + c2T(v2) + · · · + cnT(vn)
Now each vector T(vj ) is in W and therefore because γ is a basis of W there are scalars
a1,j , a2,j , . . . , am,j such that
T(vj ) = a1,j w1 + a2,j w2 + · · · + am,j wm
In other words,
[T(vj )]γ = (a1,j , a2,j , . . . , am,j )
Substituting T(vj ) = a1,j w1 + a2,j w2 + · · · + am,j wm for each j = 1, 2, . . . , n into
T(v) = c1T(v1) + c2T(v2) + · · · + cnT(vn) and then simplifying we get
m n
Σ Σ
i=1 j=1
cj ai,j
!
T(v) = wi
Therefore,
[T(v)]γ = A[v]B
where A is the m × n matrix given by
A = [T(v1)]γ [T(v2)]γ · · · [T(vn)]γ
The matrix A is the matrix representation of the linear mapping T in the bases B and γ.
Example 18.9. Consider the vector space V = P2[t] of polynomial of degree no more than two and let T : V → V be defined by
T(v(t)) = 4v′(t) − 2v(t)
It is straightforward to verify that T is a linear mapping. Let
2
B = {v1, v2, v3} = {t − 1, 3 + 2t, t + 1}.
144
Coordinate Systems
Solution. (a) Suppose that there are scalars c1, c2, c3 such that
c1v1 + c2v2 + c3v3 = 0 Then expanding and then collecting like terms we obtain
c3t2 + (c1 + 2c2)t + (−c1 + 3c2 + c3) = 0
Since the above holds for all t ∈ R we must have
c3 = 0, c1 + 2c2 = 0, −c1 + 3c2 + c3 = 0
Solving for c1, c2, c3 we obtain c1 = 0, c2 = 0, c3 = 0. Hence, the only linear combination of the vectors in B that produces the zero vector is the trivial linear combination. This proves by definition that B is linearly independent. Since we already know that dim(P2) = 3 and B contains 3 vectors, then B is a basis for P2
c1v1 + c2v2 + c3v3 = v
In this case the linear system is
c3 = −1, c1 + 2c2 = 3, −c1 + 3c2 + c3 = 1
and solving yields c1 = 1, c2 = 1, and c3 = −1. Hence,
[v]B = (1, 1, −1)
1 B
A = [T(v )] [T(v2)]B [T(v3)]B
Now we compute directly that
T(v1) = −2t + 6, T(v2) = −4t + 2, T(v3) = −2t2 + 8t − 2
And then one computes that
−18/5
[T(v1)]B = 4/5
0
,
2 B
[T(v )] = −
0
−6/5
3 B
2/5 , [T(v )] = 8/5
−2
24/5
And therefore
A =
−18/5 −6/5 24/5
0
4/5 −2/5 8/5
0 −2
145
Lecture 18
After this lecture you should know the following:
146
Coordinate Systems
Lecture 19
Change of Basis
19.1 Review of Coordinate Mappings on Rn
Let B = {v1, . . . , vn} be a basis for Rn and let
PB = [v1 v2 · · · vn].
If x ∈ Rn and [x]B is the coordinate vector of x in the basis B then
x = PB[x]B.
The components of the vector x are the coordinates of x in the standard basis E = {e1, . . . , en}. In other words,
[x]E = x.
Therefore,
[x]E = PB[x]B.
We can therefore interpret PB as the matrix mapping that maps the B-coordinates of x to the E-coordinates of x. To make this more explicit, we sometimes use the notation
E PB
to indicate that E PB maps B-coordinates to E-coordinates:
[x]E = (E PB)[x]B.
If we multiply the equation
[x]E = (E PB)[x]B
on the left by the inverse of E PB we obtain
(E PB)−1[x]E = [x]B
Hence, the matrix (E PB)−1 maps standard coordinates to B-coordinates, see Figure 19.1. It is natural then to introduce the notation
BPE = (E PB)−1
148
Change of Basis
V = Rn
x
b
BPE = (E PB)−1
[x]B
Figure 19.1: The matrix BPE maps E coordinates to B coordinates.
Example 19.1. Let
1
0
v = 0 , v
1 2
−3
0
= 4 , v
2
= −6 , x = 2 .
3 3
3
−8
Solution. Let
B
1
−3
P = 0
3
4 −6
0 0 3
It is clear that det(PB) = 12, and therefore v1, v2, v3 are linearly independent. Therefore, B is a basis for Rn. The matrix PB takes B-coordinates to standard coordinates. The B-coordinate vector [x]B = (c1, c2, c3) is the unique solution to the linear system
x = PB[x]B
Solving the linear system with augmented matrix [PB x] we obtain
[x]B = (−5, 2, 1)
We verify that [x]B = (−5, 2, 1) are indeed the coordinates of x = (−8, 2, 3) in the basis
149
Lecture 19
B = {v1, v2, v3}:
1 2 3
(−5)v + (2)v + (1)v = −
5 0 + 2 4 + −6
1 −3 3
0 0 3
−5 −6 3
= 0 + 8 + −6
0 0 3
−8
= 2
3
` ˛x¸ x
19.2 Change of Basis
We saw in the previous section that the matrix
E PB
takes as input the B-coordinates [x]B of a vector x and returns the coordinates of x in the standard basis. We now consider the situation of dealing with two basis B and C where neither is assumed to be the standard basis E. Hence let B = {v1, v2, . . . , vn} and let C = {w1, . . . , wn} be two basis of Rn and let
E PB = [v1 v2 · · · vn]
E PC = [w1 w2 · · · wn].
Then if [x]C is the coordinate vector of x in the basis C then
x = (E PC)[x]C.
How do we transform B-coordinates of x to C-coordinates of x, and vice-versa? To answer this question, start from the relations
x = (E PB)[x]B
x = (E PC)[x]C.
Then
(E PC)[x]C = (E PB)[x]B
and because E PC is invertible we have that
[x]C = (E PC)−1(E PB)[x]B.
b
[x]C
150
b
[x]B
CPB
Change of Basis
Hence, the matrix (E PC)−1(E PB) maps the B-coordinates of x to the C-coordinates of x. For this reason, it is natural to use the notation (see Figure 19.2)
CPB = (E PC)−1(E PB).
V = Rn
x
b
E PB
E PC
Figure 19.2: The matrix CPB maps B-coordinates to C-coordinates.
If we expand (E PC)−1(E PB) we obtain that
(E PC)−1(E PB) = (E PC)−1v1 (E PC)−1v2 · · · (E PC)−1vn .
Therefore, the ith column of (E PC)−1(E PB), namely
(E PC)−1vi,
is the coordinate vector of vi in the basis C = {w1, w2, . . . , wn}. To compute CPB we augment E PC and E PB and row reduce fully:
E PC E PB ~ In CPB .
Example 19.2. Let
B =
1 −2
, , C = ,
−7 −5
−3 4 9 7
It can be verified that B = {v1, v2} and C = {w1, w2} are bases for R2.
Solution. The matrix E PB = [v1 v2] maps B-coordinates to standard E-coordinates. The matrix E PC = [w1 w2] maps C-coordinates to standard E-coordinates. As we just showed, the matrix that maps B-coordinates to C-coordinates is
CPB = (E PC)−1(E PB)
151
Lecture 19
It is straightforward to compute that
−1
(E PC) =
"−7/4
9/4
−5/4#
7/4
Therefore,
−1
CPB = (E PC) (E PB) =
"
# "
−7/4 −5/4 1 −2
9/4 7/4 −3 4
# "
=
2 −3/2
−3 5/2
#
To compute BPC, we can simply invert CPB. One finds that
C B
−1
( P ) =
5 3
6 4
and therefore
B C
P =
5 3
6 4
Given that x = (0, −2), to find [x]B we must solve the linear system
E PB[x]B = x
Row reducing the augmented matrix [E PB x] we obtain
B
[x] =
2
1
Next, to find [x]C we can solve the linear system
E PC[x]C = x
Alternatively, since we now know [x]B and C PB has been computed, to find [x]C we simply multiply CPB by [x]B:
" 2 # " # "
−3/2 2 5/2
[x]C = CPB[x]B = =
−3 5/2 1 −7/2
#
Let’s verify that [x]C =
5/2
−7/2
are indeed the C-coordinates of x =
0
−2
:
E PC[x]C =
=
"−7 −5# " 5/2 0
# " #
.
9 7 −7/2 −2
After this lecture you should know the following:
152
Change of Basis
Lecture 20
153
Lecture 20
Inner Products and Orthogonality
20.1 Inner Product on Rn
The inner product on Rn generalizes the notion of the dot product of vectors in R2 and R3
that you may are already familiar with.
Definition 20.1: Let u = (u1, u2, . . . , un) and let v = (v1, v2, . . . , vn) be vectors in Rn.
The inner product of u and v is
u • v = u1v1 + u2v2 + · · · + unvn.
Notice that the inner product u • v can be computed as a matrix multiplication as follows:
v1
•
1
2
u v = uT v = u u · · · u
n
v
2
.
.
.
vn
The following theorem summarizes the basic algebraic properties of the inner product.
Theorem 20.2: Let u, v, w be vectors in Rn and let α be a scalar. Then
154
Inner Products and Orthogonality
Example 20.3. Let u = (2, −5, −1) and let v = (3, 2, −3). Compute u • v, v • u, u • u, and
v • v.
Solution. By definition:
u • v = (2)(3) + (−5)(2) + (1)(−3) = −1
v • u = (3)(2) + (2)(−5) + (−3)(1) = −1
u • u = (2)(2) + (−5)(−5) + (−1)(−1) = 30
v • v = (3)(3) + (2)(2) + (−3)(−3) = 22.
We now define the length or norm of a vector in Rn.
Definition 20.4: The length or norm of a vector u ∈ Rn is defined as
√
•
q
2
1
2
2
2
n
u = u u = u + u + · · · + u .
A vector u ∈ Rn with norm 1 will be called a unit vector:
u = 1.
Below is an important property of the inner product.
Theorem 20.5: Let u ∈ Rn and let α be a scalar. Then
αu = |α| u .
Proof. We have
αu = √(αu) • (αu)
= √α2(u • u)
= |α|√u • u
= |α| u .
By Theorem 20.5, any non-zero vector u ∈ Rn can be scaled to obtain a new unit vector in the same direction as u. Indeed, suppose that u is non-zero so that u =/ 0. Define the
new vector
v =
1
u
u
155
Lecture 20
u
Notice that α = 1 is just a scalar and thus v is a scalar multiple of u. Then by Theorem 20.5
we have that
v = αu = |α| · u =
1
u
· u = 1
and therefore v is a unit vector, see Figure 20.1. The process of taking a non-zero vector u
u
and creating the new vector v = 1 u is sometimes called normalization of u.
u
u
v = 1 u
Figure 20.1: Normalizing a non-zero vector.
Example 20.6. Let u = (2, 3, 6). Compute u and find the unit vector v in the same direction as u.
Solution. By definition,
u = √u • u = √22 + 32 + 62 = √49 = 7.
Then the unit vector that is in the same direction as u is
v =
1
u
u =
1
7
2 2/7
6
3 = 3/7
6/7
Verify that v = 1:
v = √(2/7)2 + (3/7)2 + (6/7)2 = √4/49 + 9/49 + 36/49 = √49/49 = √1 = 1.
Now that we have the definition of the length of a vector, we can define the notion of distance between two vectors.
Definition 20.7: Let u and v be vectors in Rn. The distance between u and v is the length of the vector u − v. We will denote the distance between u and v by d(u, v). In other words,
d(u, v) = u − v .
Example 20.8. Find the distance between u =
3
−2
and v =
−9
7 .
Solution. We compute:
d(u, v) = u − v = √(3 − 7)2 + (−2 + 9)2 = √65.
156
Inner Products and Orthogonality
20.2 Orthogonality
In the context of vectors in R2 and R3, orthogonality is synonymous with perpendicularity. Below is the general definition.
Definition 20.9: Two vectors u and v in Rn are said to be orthogonal if u • v = 0.
In R2 and R3, the notion of orthogonality should be familiar to you. In fact, using the Law of Cosines in R2 or R3, one can prove that
u • v = u · v cos(θ) (20.1)
π
where θ is the angle between u and v. If θ = 2 then clearly u • v = 0. In higher dimensions,
i.e., n ≥ 4, we can use equation (20.1) to define the angle between vectors u and v. In other words, the angle between any two vectors u and v in Rn is define to be
θ = arccos
u
u · v
.
The general notion of orthogonality in Rn leads to the following theorem from grade school.
Theorem 20.10: (Pythagorean Theorem) Two vectors u and v are orthogonal if and only if u + v 2 = u 2 + v 2.
Solution. First recall that u + v = √(u + v) • (u + v) and therefore
2
u + v = (u + v) • (u + v)
= u • u + u • v + v • u + v • v
= u 2 + 2(u • v) + v 2.
Therefore, u + v 2 = u 2 + v 2 if and only if u • v = 0.
We now introduce orthogonal sets.
Definition 20.11: A set of vectors {u1, u2, . . . , up} is said to be an orthogonal set if any pair of distinct vectors ui, uj are orthogonal, that is, ui • uj = 0 whenever i /= j.
In the following theorem we prove that orthogonal sets are linearly independent.
157
Lecture 20
Theorem 20.12: Let {u1, u2, . . . , up} be an orthogonal set of non-zero vectors in Rn.
1 2 p
Then the set {u , u , . . . , u } is linearly independent. In particular, if p = n then the set
1 2 n
n
{u , u , . . . , u } is basis for R .
Solution. Suppose that there are scalars c1, c2, . . . , cp such that
c1u1 + c2u2 + · · · + cpup = 0.
Take the inner product of u1 with both sides of the above equation:
c1(u1 • u1) + c2(u2 • u1) + · · · + cp(up • u1) = 0 • u1.
Since the set is orthogonal, the left-hand side of the last equation simplifies to c1(u1 • u1). The right-hand side simplifies to 0. Hence,
c1(u1 • u1) = 0.
2
But u1 • u1 = u1 is not zero and therefore the only way that c1(u1 • u2) = 0 is if c1 = 0. Repeat the above steps using u2, u3, . . . , up and conclude that c2 = 0, c3 = 0, . . . , cp =
0. Therefore, {u1, . . . , up} is linearly independent. If p = n, then the set {u1, . . . , up} is automatically a basis for Rn.
Example 20.13. Is the set {u1, u2, u3} an orthogonal set?
1 0 −5
1
u = −
1
2
2 3
2 , u = 1 , u = −
2
1
Solution. Compute
u1 • u2 = (1)(0) + (−2)(1) + (1)(2) = 0
u1 • u3 = (1)(−5) + (−2)(−2) + (1)(1) = 0
u2 • u3 = (0)(−5) + (1)(−2) + (2)(1) = 0
1 2 3
1 2 3
Therefore, {u , u , u } is an orthogonal set. By Theorem 20.12, the set {u , u , u } is linearly
independent. To verify linear independence, we computed that det( u1
u2 u3 ) = 30,
which is non-zero.
158
Inner Products and Orthogonality
We now introduce orthonormal sets.
Definition 20.14: A set of vectors {u1, u2, . . . , up} is said to be an orthonormal set if it is an orthogonal set and if each vector ui in the set is a unit vector.
Consider the previous orthogonal set in R3:
1 2 3
1
{u , u , u } = −
1 2 1
0 −5
2 , 1 , −2 .
1 2 3
It is not an orthonormal set because none of u , u , u are unit vectors. Explicitly, u1 =
√ √ √
6, u2 = 5, and u3 = 30. However, from an orthogonal set we can create an
orthonormal set by normalizing each vector. Hence, the set
1 2 3
{v , v , v } = −
√
√
1/ 6
√
√
2/ 5
1/ 6 0 −5/ 30
2/ 6 , 1/ 5 , −
√
2/ 30
√
1/ 30
√ √
is an orthonormal set.
20.3 Coordinates in an Orthonormal Basis
As we will see in this section, a basis B = {u1, u2, . . . , un} of Rn that is also an orthonormal set is highly desirable when performing computations with coordinates. To see why, let x be any vector in Rn and suppose we want to find the coordinates of x in the basis B, that is we seek to find [x]B = (c1, c2, . . . , cn). By definition, the coordinates c1, c2, . . . , cn satisfy the
equation
x = c1u1 + c2u2 + · · · + cnun.
Taking the inner product of u1 with both sides of the above equation and using the fact that
u1 • u2 = 0, u1 • u3 = 0, and u1 • un = 0, we obtain
u1 • x = c1(u1 • u1) = c1(1) = c1
where we also used the fact that ui is a unit vector. Thus, c1 = u1 • x! Repeating this procedure with u2, u3, . . . , un we obtain the remaining coefficients c2, . . . , cn:
c2 = u2 • x
c3 = u3 • x
.. = ..
cn = un • x.
Our previous computation proves the following theorem.
159
B
[x] =
2
•
u x
.
.
un • x
Lecture 20
Theorem 20.15: Let B = {u1, u2, . . . , un} be an orthonormal basis for Rn. The coordi- nate vector of x in the basis B is
u1 • x
.
B
[x] =
1
•
2
•
u x
un • x
u x u
T
1
u
. .
. .
uT
n
Hence, computing coordinates with respect to an orthonormal basis can be done without performing any row operations and all we need to do is compute inner products! We make the important observation that an alternate expression for [x]B is
T
2
=
T
x = U x
where U = [u1 u2 · · · un]. On the other hand, recall that by definition [x]B satisfies
U[x]B = x, and therefore [x]B = U−1x. If we compare the two identities
[x]B = U−1x and [x]B = UT x
we suspect then that U−1 = UT . This is indeed the case. To see this, let B = {u1, u2, . . . , un}
be an orthonormal basis for Rn and put
i
U = [u1 u2 · · · un].
Consider the matrix product UT U, and recalling that ui • uj = uT uj , we obtain
T
u
T
1
u
.
.
uT
n
T
2
U U =
1
2
u u · · · u
n
=
T
1
u u
1
1
T
1
n
T
2
u u
1
2
uT u2
T
2
uT u2 · · · u u
· · · u u
n
.
.
.
.
. .
.
.
T
n
u u
1
n
T
n
uT u2 · · · u u
n
.
= In.
160
Inner Products and Orthogonality
Therefore,
U−1 = UT .
A matrix U ∈ Rn×n such that
UT U = UUT = In
is called a orthogonal matrix. Hence, if B = {u1, u2, . . . , un} is an orthonormal set then
the matrix
U = u1 u2 · · ·
un
is an orthogonal matrix.
Example 20.16. Consider the vectors
1 −1
v = 0 , v
1 2
= 4 , v
3
1 1 −2
2
1
−1
= 1 , x = 2 .
{u1, u2, u3} for R3.
Solution. (a) We compute that v1 • v2 = 0, v1 • v3 = 0, and v2 • v3 = 0, and thus {v1, v2, v3}
is an orthogonal set. Since orthogonal sets are linearly independent and {v1, v2, v3}
1 2 3
3
consists of three vectors then {v , v , v } is basis for R .
√
(b) We compute that v1 = 2, v2
√
= 18, and v3 = 3. Then let
u1 = 0
1/√2
−1/ 18
√
√ √
1/ 2 1/ 18
√
, u = 4/ 18 , u
2 3
= 1/3
−2/3
2/3
Then B = {u1, u2, u3} is now an orthonormal set and thus since B consists of three vectors then B is an orthonormal basis of R3.
(c) Finally, computing coordinates in an orthonormal basis is easy:
u1 • x √0
B 2
•
u3 • x
[x] = u x = 2/ 18
5/3
Example 20.17. The standard unit basis
1 2 3
E = {e , e , e } =
0
0
0 , 1 , 0
1
1 0 0
161
Lecture 20
in R3 is an orthonormal basis. Given any x = (x1, x2, x3), we have [x]E = x. On the other hand, clearly
x1 = x • e1 x2 = x • e2 x3 = x • e3
Example 20.18. (Orthogonal Complements) Let W be a subspace of Rn. The orthogonal complement of W, which we denote by W⊥, consists of the vectors in Rn that are orthogonal to every vector in W. Using set notation:
W⊥ = {u ∈ Rn : u • w = 0 for every w ∈ W}.
W⊥.
Solution. (a) The vector 0 is orthogonal to every vector in Rn and therefore it is certainly orthogonal to every vector in W. Thus, 0 ∈ W⊥. Now suppose that u1, u2 are two vectors in W⊥. Then for any vector w ∈ W it holds that
(u1 + u2) • w = u1 • w + u2 • w = 0 + 0 = 0.
Therefore, u1 + u2 is also orthogonal to w and since w is an arbitrary vector in W then (u1 + u2) ∈ W⊥. Lastly, let α be any scalar and let u ∈ W⊥. Then for any vector w in W we have that
(αu) • w = α(u • w) = α · 0 = 0.
Therefore, αu is orthogonal to w and since w is an arbitrary vector in W then (αu) ∈ W⊥.
This proves that W⊥ is a subspace of Rn.
(b) A vector u = (u1, u2, u3, u3) is in W⊥ if u • w1 = 0 and u • w2 = 0. In other words, if
u2 + u3 = 0
u1 − u3 = 0
This is a linear system for the unknowns u1, u2, u3, u4. The general solution to the linear system is
u = t
1 0
0
1
1 −1
+ s .
0 0
Therefore, a basis for W⊥ is {(1, 0, 1, 0), (0, 1, −1, 0)}.
After this lecture you should know the following:
162
Inner Products and Orthogonality
Lecture 21
163
Lecture 21
Eigenvalues and Eigenvectors
21.1 Eigenvectors and Eigenvalues
An n × n matrix A can be thought of as the linear mapping that takes any arbitrary vector x ∈ Rn and outputs a new vector Ax. In some cases, the new output vector Ax is simply a scalar multiple of the input vector x, that is, there exists a scalar λ such that Ax = λx. This case is so important that we make the following definition.
Definition 21.1: Let A be a n × n matrix and let v be a non-zero vector. If Av = λv for some scalar λ then we call the vector v an eigenvector of A and we call the scalar λ an eigenvalue of A corresponding to v.
Hence, an eigenvector v of A is simply scaled by a scalar λ under multiplication by A. Eigenvectors are by definition nonzero vectors because A0 is clearly a scalar multiple of 0 and then it is not clear what that the corresponding eigenvalue should be.
Example 21.2. Determine if the given vectors v and u are eigenvectors of A? If yes, find the eigenvalue of A associated to the eigenvector.
−1
2 −1 8
4 6
A = 2 1 6 ,
−3 −1
1
v = 0 ,
1
u = 2 .
Solution. Compute
4
−1
Av = 2
6 −3 −6
1 6 0 = 0
2 −1 8 1
2
= 2 0
−3
1
= 2v
164
Eigenvalues and Eigenvectors
Hence, Av = 2v and thus v is an eigenvector of A with corresponding eigenvalue λ = 2. On the other hand,
−1
Au = 2
2 −1 8 1
4
4 6 −1 0
1 6 2 = 6 .
There is no scalar λ such that
0 −1
4
1
6 = λ 2 .
Therefore, u is not an eigenvector of A.
Example 21.3. Is v an eigenvector of A? If yes, find the eigenvalue of A associated to v:
A =
−1 2 −
−4 2 2
−1 −1
1
2 1
1 , v = 1 .
Solution. We compute
0
Av = 0 = 0.
0
Hence, if λ = 0 then λv = 0 and thus Av = λv. Therefore, v is an eigenvector of A with corresponding eigenvalue λ = 0.
How does one find the eigenvectors/eigenvalues of a matrix A? The general procedure is to first find the eigenvalues of A and then for each eigenvalue find the corresponding eigenvectors. In this section, however, we will instead suppose that we have already found the eigenvalues of A and concern ourselves with finding the associated eigenvectors. Suppose then that λ is known to be an eigenvalue of A. How do we find an eigenvector v corresponding to the eigenvalue λ? To answer this question, we note that if v is to be an eigenvector of A with eigenvalue λ then v must satisfy the equation
Av = λv.
We can rewrite this equation as
Av − λv = 0
which, after using the distributive property of matrix multiplication, is equivalent to
(A − λI)v = 0.
The last equation says that if v is to be an eigenvector of A with eigenvalue λ then v must be in the null space of A − λI:
v ∈ Null(A − λI).
165
Lecture 21
In summary, if λ is known to be an eigenvalue of A, then to find the eigenvectors corre- sponding to λ we must solve the homogeneous system
(A − λI)x = 0.
Recall that the null space of any matrix is a subspace and for this reason we call the subspace Null(A − λI) the eigenspace of A corresponding to λ.
Example 21.4. It is known that λ = 4 is an eigenvalue of
−4 6
3
A = 1 7 9 .
8 −6 1
Find a basis for the eigenspace of A corresponding to λ = 4.
Solution. First compute
−4 6
A − 4I = 1 7 9
8 −6 1
3 4
−8 6
0 0
— 0 4 0 = 1 3 9
0 0 4 8 −6 −3
3
Find a basis for the null space of A − 4I:
3
R1‡R2
9
9
8R1 +R2
−8R1 +R3
1
3 −−−−−−→ 0
75
9
Finally,
1
R2 +R3
9 1
0 30 75 −−−−→ 0
75
0 −30 −75 0
−8 6 1 3
1 3 9 −−−→ −8 6 3
8 −6 −3 8 −6 −3
1 3 3
−8 6 30
8 −6 −3 0 −30 −75
3 3
30
0
0
9
x = t −5/2
Hence, the general solution to the homogenous system (A − 4I)x = 0 is
−3/2
1
where t is an arbitrary scalar. Therefore, the eigenspace of A corresponding to λ = 4 is
span −
−3/2
5/2 = span −
1 2
−3
5 = span{v}
and {v} is a basis for the eigenspace. The vector v is of course an eigenvector of A with eigenvalue λ = 4 and also (of course) any multiple of v is also eigenvector of A with λ = 4.
166
Eigenvalues and Eigenvectors
Example 21.5. It is known that λ = 3 is an eigenvalue of
11 −4 −8
A = 4 1 −4 .
8 −4 −5
Find the eigenspace of A corresponding to λ = 3.
Solution. First compute
11 −4 −8 3 0
−4 − 0
A − 3I = 4 1
8 −4 −5 0
0 8
3 0 = 4
−4 −8
−2 −4
0 3 8 −4 −8
Now find the null space of A − 3I:
8
−4 −8
R1‡R2
4
4 −2 −4 −−−→ 8 −4 −8
8 −4 −8 8 −4 −8
−2 −4
4
−2 −4 −2R1 +R2
−2R1 +R3
4
8 −4 −8 −−−−−−→ 0 0 0
8 −4 −8 0 0 0
−2 −4
Hence, any vector in the null space of
A − 3I =
4
0 0 0
−2 −4
0 0 0
1 1
can be written as
1 2
x = t 0 + t 2
1 0
Therefore, the eigenspace of A corresponding to λ = 3 is
Null(A − 3I) = span
1 2
{v , v } = span 0 , 2
1 0
1 1
.
The vectors v1 and v2 are two linearly independent eigenvectors of A with eigenvalue λ = 3. Therefore {v1, v2} is a basis for the eigenspace of A with eigenvalue λ = 3. You can verify that Av1 = 3v1 and Av2 = 3v2.
As shown in the last example, there may exist more than one linearly independent eigen- vector of A corresponding to the same eigenvalue, in other words, it is possible that the dimension of the eigenspace Null(A − λI) is greater than one. What can be said about the eigenvectors of A corresponding to different eigenvalues?
167
Lecture 21
Theorem 21.6: Let v1, . . . , vk be eigenvectors of A corresponding to distinct eigenvalues
λ1, . . . , λk of A. Then {v1, . . . , vk} is a linearly independent set.
Solution. Suppose by contradiction that {v1, . . . , vk} is linearly dependent and {λ1, . . . , λk} are distinct. Then, one of the eigenvectors vp+1 that is a linear combination of v1, . . . , vp, and {v1, . . . , vp} is linearly independent:
vp+1 = c1v1 + c2v2 + · · · + cpvp. (21.1) Applying A to both sides we obtain
Avp+1 = c1Av1 + c2Av2 + · · · + cpAvp
and since Avi = λivi we can simplify this to
λp+1vp+1 = c1λ1v1 + c2λ2v2 + · · · + cpλpvp.
On the other hand, multiply (21.1) by λp+1:
(21.2)
(21.3)
0 = c1(λ1 − λp+1)v1 + c2(λ2 − λp+1)v2 + · · · + cp(λp − λp+1)vp.
Now {v1, . . . , vp} is linearly independent and thus ci(λi − λp+1) = 0. But the eigenvalues
{λ1, . . . , λk} are all distinct and so we must have c1 = c2 = · · · = cp = 0. But from (21.1) this implies that vp+1 = 0, which is a contradiction because eigenvectors are by definition non-zero. This proves that {v1, v2, . . . , vk} is a linearly independent set.
Example 21.7. It is known that λ1 = 1 and λ2 = −1 are eigenvalues of
−4 6
3
A = 1 7 9 .
8 −6 1
Find bases for the eigenspaces corresponding to λ1 and λ2 and show that any two vectors from these distinct eigenspaces are linearly independent.
Solution. Compute
−5 6
A − λ1I = 1 6 9
8 −6 0
3
and one finds that
(A − λ1I) = span
−4
3
−3
168
Eigenvalues and Eigenvectors
Hence, v1 = (−3, −4, 3) is an eigenvector of A with eigenvalue λ1 = 1, and {v1} forms a basis for the corresponding eigenspace. Next, compute
−4 6
3 1
A − λ2I = 1 7 9 + 0
0 0
1 0 =
8 −6 1 0 0 1
−3 6
3
1 8 9
8 −6 2
and one finds that
−1
A − λ2I = span −
1
1
Hence, v2 = (−1, −1, 1) is an eigenvector of A with eigenvalue λ2 = −1, and {v2} forms a basis for the corresponding eigenspace. Now verify that v1 and v2 are linearly independent:
v1 v2 = −
−3 −1
R1 +R3
4 −1 −−−−→ −4 −1
−3 −1
3 1 0 0
The last matrix has rank r = 2, and thus v1, v2 are indeed linearly independent.
What can we say about A if λ = 0 is an eigenvalue of A? Suppose then that A has eigenvalue
λ = 0. Then by definition, there exists a non-zero vector v such that
Av = 0 · v = 0.
In other words, v is in the null space of A. Thus, A is not invertible (Why?).
Theorem 21.8: The matrix A ∈ Rn×n is invertible if and only if λ = 0 is not an eigenvalue of A.
In fact, later we will see that det(A) is the product of its eigenvalues.
After this lecture you should know the following:
Lecture 22
169
Lecture 22
The Characteristic Polynomial
22.1 The Characteristic Polynomial of a Matrix
Recall that a number λ is an eigenvalue of A ∈ Rn×n if there exists a non-zero vector v such that
Av = λv
or equivalently if v ∈ Null(A − λI). In other words, λ is an eigenvalue of A if and only if the subspace Null(A − λI) contains a vector other than the zero vector. We know that any matrix M has a non-trivial null space if and only if M is non-invertible if and only if det(M) = 0. Hence, λ is an eigenvalue of A if and only if λ satisfies det(A − λI) = 0. Let’s compute the expression det(A − λI) for a generic 2 × 2 matrix:
det(A − λI) =
11
a − λ
a12
a21
a22 − λ
= (a11 − λ)(a22 − λ) − a12a22
= λ2 − (a11 + a22)λ + a11a22 − a12a22.
Thus, if A is 2 × 2 then
det(A − λI) = λ2 − (a11 + a22)λ + a11a22 − a12a22
is a polynomial in the variable λ of degree n = 2. This motivates the following definition.
Definition 22.1: Let A be a n × n matrix. The polynomial
p(λ) = det(A − λI) is called the characteristic polynomial of A.
170
The Characteristic Polynomial
In summary, to find the eigenvalues of A we must find the roots of the characteristic poly- nomial:
p(λ) = det(A − λI).
The following theorem asserts that what we observed for the case n = 2 is indeed true for all n.
Theorem 22.2: The characteristic polynomial p(λ) = det(A − λI) of a n × n matrix A
is an nth degree polynomial.
Solution. Recall that for the case n = 2 we computed that
11 11
2
det(A − λI) = λ − (a11 + a22)λ + a11a22 − a12a22.
Therefore, the claim holds for n = 2. By induction, suppose that the claims hold for n ≥ 2. If A is a (n + 1) × (n + 1) matrix then expanding det(A − λI) along the first row:
n
Σ
det(A − λI) = (a − λ) det(A − λI) + (−
1+k
1) a det(A
1k 1k
— λI).
k=2
By induction, each of det(A1k −λI) is a nth degree polynomial. Hence, (a11 −λ) det(A11 −λI) is a (n + 1)th degree polynomial. This ends the proof.
Example 22.3. Find the characteristic polynomial of
A =
−2 4
−6 8
.
What are the eigenvalues of A?
Solution. Compute
A − λI =
−2 4
−6 8
—
λ 0
0 λ
=
−2 − λ 4
−6 8 − λ
.
Therefore,
p(λ) = det(A − λI)
=
−2 − λ 4
−6 8 − λ
= (−2 − λ)(8 − λ) + 24
= λ2 − 6λ + 8
= (λ − 4)(λ − 2)
The roots of p(λ) are clearly λ1 = 4 and λ2 = 2. Therefore, the eigenvalues of A are λ1 = 4 and λ2 = 2.
171
Lecture 22
Example 22.4. Find the eigenvalues of
0 0 3
−4 −6 −7
A = 3 5 3 .
Solution. Compute
0
=
−4 − λ
−4 −6 −7 λ 0 0
A − λI = 3 5 3 − 0 λ
0 0 3 0 0 λ
−6 −7
3 5 − λ 3
0 0 3 − λ
Then
det(A − λI) = (−4 − λ)
5 − λ 3
— 3
−6 −7
−λ 3 − λ −λ 3 − λ
= (−4 − λ)[(3 − λ)(5 − λ) + 3λ] − 3[−6(3 − λ) − 7λ]
= λ3 − 4λ2 + λ + 6
Factor the characteristic polynomial:
p(λ) = λ3 − 4λ2 + λ + 6 = (λ − 2)(λ − 3)(λ + 1)
Therefore, the eigenvalues of A are
λ1 = 2, λ2 = 3, λ3 = −1.
Now that we know how to find eigenvalues, we can combine our work from the previous lecture to find both the eigenvalues and eigenvectors of a given matrix A.
Example 22.5. For each eigenvalue of A from Example 22.4, find a basis for the corre- sponding eigenspace.
Solution. Start with λ1 = 2:
A − 2I = 3 3 3
0 0 1
−6 −6 −7
After basic row reduction and back substitution, one finds that the null space of A − 2I is
spanned by
1
v =
−1
0
1 .
172
The Characteristic Polynomial
Therefore, v1 is an eigenvector of A with eigenvalue λ1. For λ2 = 3:
A − 3I = 3 2 3
0 0 0
−7 −6 −7
The null space of A − 3I is spanned by
−1
2
v = 0
1
and therefore v2 is an eigenvector of A with eigenvalue λ2. Finally, for λ3 = −1 we compute
A − λ3I = 3 6 3
0 0 4
−3 −6 −7
and the null space of A − λ3I is spanned by
−2
3
v = 1
0
and therefore v3 is an eigenvector of A with eigenvalue λ3. Notice that in this case, the 3 × 3 matrix A has three distinct eigenvalues and the eigenvectors
−1 −1 −2
1 2 3
{v , v , v } =
1 , 0 , 1
0 1 0
correspond to the distinct eigenvalues λ1, λ2, λ3, respectively. Therefore, the set β = {v1, v2, v3} is linearly independent (by Theorem 21.6), and therefore β is a basis for R3. You can verify, for instance, that det([v1 v2 v3]) /= 0.
By Theorem 21.6, the previous example has the following generalization.
Theorem 22.6: Suppose that A is a n × n matrix and has n distinct eigenvalues λ1, λ2, . . . , λn. Let vi be an eigenvector of A corresponding to λi. Then {v1, v2, . . . , vn} is a basis for Rn.
Hence, if A has distinct eigenvalues, we are guaranteed the existence of a basis of Rn consisting of eigenvectors of A. In forthcoming lectures, we will see that it is very convenient to work with matrices A that have a set of eigenvectors that form a basis of Rn; this is one of the main motivations for studying eigenvalues and eigenvectors in the first place. However, we will see that not every matrix has a set of eigenvectors that form a basis of Rn. For example, what if A does not have n distinct eigenvalues? In this case, does there exist a
173
Lecture 22
basis for Rn of eigenvectors of A? In some cases, the answer is yes as the next example demonstrates.
Example 22.7. Find the eigenvalues of A and a basis for each eigenspace.
2 0
A = 4 2 2
−2 0 1
0
Does R3 have a basis of eigenvectors of A?
Solution. The characteristic polynomial of A is
p(λ) = det(A − λI) = λ3 − 5λ2 + 8λ − 4 = (λ − 1)(λ − 2)2
and therefore the eigenvalues are λ1 = 1 and λ2 = 2. Notice that although p(λ) is a polynomial of degree n = 3, it has only two distinct roots and hence A has only two distinct eigenvalues. The eigenvalue λ2 = 2 is said to be repeated and λ1 = 1 is said to be a simple eigenvalue. For λ1 = 1 one finds that the eigenspace Null(A − λ1I) is spanned by
0
1
v = −
2
1
and thus v1 is an eigenvector of A with eigenvalue λ1 = 1. Now consider λ2 = 2:
0 0
A − 2I = 4 0 2
−2 0 −1
0
Row reducing A − 2I one obtains
A − 2I =
0 −1 0 0 0
0 0 0 −2 0 −1
4 0 2 ~ 0 0 0 .
−2
Therefore, rank(A − 2I) = 1 and thus by the Rank Theorem it follows that Null(A − 2I) is
a 2-dimensional eigenspace. Performing back substitution, one finds the following basis for the λ2-eigenspace:
2 3
{v , v } =
−1 0
0 , 1
2 0
Therefore, the eigenvectors
1 2 3
0
{v , v , v } = −
2 , 0 , 1
1 2 0
−1 0
form a basis for R3. Hence, for the repeated eigenvalue λ2 = 2 we were able to find two
linearly independent eigenvectors.
174
The Characteristic Polynomial
Before moving further with more examples, we need to introduce some notation regard- ing the factorization of the characteristic polynomial. In the previous Example 22.7, the characteristic polynomial was factored as p(λ) = (λ − 1)(λ − 2)2 and we found a basis for R3 of eigenvectors despite the presence of a repeated eigenvalue. In general, if p(λ) is an nth degree polynomial that can be completely factored into linear terms, then p(λ) can be written in the form
p(λ) = (λ − λ1)k1 (λ − λ2)k2 · · · (λ − λp)kp
where k1, k2, . . . , kp are positive integers and the roots of p(λ) are then λ1, λ2, . . . , λk. Because p(λ) is of degree n, we must have that k1 +k2 +· · · +kp = n. Motivated by this, we introduce the following definition.
Definition 22.8: Suppose that A ∈ Mn×n has characteristic polynomial p(λ) that can be factored as
p(λ) = (λ − λ1)k1 (λ − λ2)k2 · · · (λ − λp)kp .
The exponent ki is called the algebraic multiplicity of the eigenvalue λi. The dimension Null(A − λiI) of the eigenspace associated to λi is called the geometric multiplicity of λi.
For simplicity and whenever it is convenient, we will denote the geometric multiplicity of the eigenvalue λi as
gi = dim(Null(A − λiI)).
Example 22.9. A 6 × 6 matrix A has characteristic polynomial
p(λ) = λ6 − 4λ5 − 12λ4.
Find the eigenvalues of A and their algebraic multiplicities.
Solution. Factoring p(λ) we obtain
p(λ) = λ4(λ2 − 4λ − 12) = λ4(λ − 6)(λ + 2)
Therefore, the eigenvalues of A are λ1 = 0, λ2 = 6, and λ3 = −2. Their algebraic multiplic- ities are k1 = 4, k2 = 1, and k3 = 1, respectively. The eigenvalue λ1 = 0 is repeated, while λ2 = 6 and λ3 = −2 are simple eigenvalues.
In Example 22.7, we had p(λ) = (λ−1)(λ−2)2 and thus λ1 = 1 has algebraic multiplicity k1 = 1 and λ2 = 2 has algebraic multiplicity k2 = 2. For λ1 = 1, we found one linearly independent eigenvector, and therefore λ1 has geometric multiplicity g1 = 1. For λ1 = 2, we found two linearly independent eigenvectors, and therefore λ2 has geometric multiplicity g2 = 2. However, as we will see in the next example, the geometric multiplicity gi is in general less than the algebraic multiplicity ki:
gi ≤ ki
175
Lecture 22
Example 22.10. Find the eigenvalues of A and a basis for each eigenspace:
4
A = −4 −6 −
3
2 3
3 3 1
For each eigenvalue of A, find its algebraic and geometric multiplicity. Does R3 have a basis of eigenvectors of A?
Solution. One computes
p(λ) = −λ3 − 3λ2 + 4 = −(λ − 1)(λ + 2)2
and therefore the eigenvalues of A are λ1 = 1 and λ2 = −2. The algebraic multiplicity of λ1
is k1 = 1 and that of λ2 is k2 = 2. For λ1 = 1 we compute
4
A − I = −
4 −7 −3
3 3 0
1 3
and then one finds that
1
v = −
1
1
1
is a basis for the λ1-eigenspace. Therefore, the geometric multiplicity of λ1 is g1 =. For
2
λ2 = −2 we compute
A − λ I =
−4 −4 −
4 3 4 3 1
4 4 1
3 ~ 1 1 1 ~ 0 0 1
1
3 3 3 0 0 0 0 0 0
Therefore, since rank(A − λ2I) = 2, the geometric multiplicity of λ2 = −2 is g2 = 1, which
2
v = 1
is less than the algebraic multiplicity k2 = 2. An eigenvector corresponding to λ2 = −2 is
−1
0
Therefore, for the repeated eigenvalue λ2 = −2, we are able to find only one linearly inde-
pendent eigenvector. Therefore, it is not possible to construct a basis for R3 consisting of eigenvectors of A.
Hence, in the previous example, there does not exist a basis of R3 of eigenvectors of A because for one of the eigenvalues (namely λ2) the geometric multiplicity was less than the algebraic multiplicity:
g2 < d2.
In the next lecture, we will elaborate on this situation further.
0
0
−7
Example 22.11. Find the algebraic and geometric multiplicities of each eigenvalue of the matrix
A = 0
0
−7 1
−7 1 .
176
The Characteristic Polynomial
To end this lecture, we will define a notion of similarity between matrices that plays an important role in linear algebra and that will be used in the next lecture when we dis- cuss diagonalization of matrices. In mathematics, there are many cases where one is inter- ested in classifying objects into categories or classes. Classifying mathematical objects into classes/categories is similar to how some physical objects are classified. For example, all fruits are classified into categories: apples, pears, bananas, oranges, avocados, etc. Given a piece of fruit A, how do you decide what category it is in? What are the properties that uniquely classify the piece of fruit A? In linear algebra, there are many objects of interest. We have spent a lot of time working with matrices and we have now reached a point in our study where we would like to begin classifying matrices. How should we decide if matrices A and B are of the same type or, in other words, are similar? Below is how we will decide.
Definition 22.12: Let A and B be n × n matrices. We will say that A is similar to B
if there exists an invertible matrix P such that
A = PBP−1.
If A is similar to B then B is similar to A because from the equation A = PBP−1 we can multiply on the left by P−1 and on the right by P to obtain that
P−1AP = B.
Hence, with Q = P−1, we have that B = QAQ−1 and thus B is similar to A. Hence, if A is similar to B then B is similar to A and therefore we simply say that A and B are similar. Matrices that are similar are clearly not necessarily equal. However, there is a reason why the word similar is used. Here are a few reasons why.
Theorem 22.13: If A and B are similar matrices then the following are true:
Proof. We will prove part (c). If A and B are similar then A = PAP−1 for some matrix P.
Then
det(A − λI) = det(A − λPP−1)
= det(PBP−1 − λPP−1)
= det(P(B − λI)P−1)
= det(P) det(B − λI) det(P−1)
= det(B − λI)
177
Lecture 22
Thus, A and B have the same characteristic polynomial, and hence the same eigenvalues.
In the next lecture, we will see that if Rn has a basis of eigenvectors of A then A is similar to a diagonal matrix.
After this lecture you should know the following:
n
sisting of the eigenvectors of A
178
The Characteristic Polynomial
Lecture 23
Diagonalization
A = 0
a
a22 23
0 0
a33
23.1 Eigenvalues of Triangular Matrices
Before discussing diagonalization, we first consider the eigenvalues of triangular matrices.
Theorem 23.1: Let A be a triangular matrix (either upper or lower). Then the eigen- values of A are its diagonal entries.
Proof. We will prove the theorem for the case n = 3 and A is upper triangular; the general case is similar. Suppose then that A is a 3 × 3 upper triangular matrix:
a11 a12 a13
Then
11
a − λ
a12
a13 a23
A − λI = 0 a22 − λ
0
0 a33 − λ
.
and thus the characteristic polynomial of A is
p(λ) = det(A − λI) = (a11 − λ)(a22 − λ)(a33 − λ) and the roots of p(λ) are
λ1 = a11, λ2 = a22, λ3 = a33.
In other words, the eigenvalues of A are simply the diagonal entries of A.
Example 23.2. Consider the following matrix
6
0 | 0 | 0 |
0 | 0 | 0 |
0 | 7 | 0 |
0 | 0 | −4 |
−2 | 3 | 0 |
0
−1
A = 0
−1
0
8
7
0
0 .
180
Diagonalization
We now introduce a very special type of a triangular matrix, namely, a diagonal matrix.
Definition 23.3: A matrix D whose off-diagonal entries are all zero is called a diagonal matrix.
For example, here is 3 × 3 diagonal matrix
3
0
D = 0
0
−5 0 .
0 0 −8
and here is a 5 × 5 diagonal matrix
6
2
11
0 0 0 0
0 0 0 0 0
D = 0 0 − 7 0 0 .
0 0 0 2 0
1
0 0 0 0 −
A diagonal matrix is clearly also a triangular matrix and therefore the eigenvalues of a
diagonal matrix D are simply the diagonal entries of D. Moreover, the powers of a diagonal
matrix are easy to compute. For example, if D =
1
λ 0
0 λ2
then
2
D =
1 1
λ 0 λ 0
0 λ2 0 λ2
=
2
λ 0
2
1
0 λ2
and similarly for any integer k = 1, 2, 3, . . ., we have that
k
D =
k
λ 0
1
0
λk
2
.
23.2 Diagonalization
Recall that two matrices A and B are said to be similar if there exists an invertible matrix
P such that
A = PBP−1.
A very simple type of matrix is a diagonal matrix since many computations with diagonal matrices are trivial. The problem of diagonalization is thus concerned with answering the question of whether a given matrix is similar to a diagonal matrix. Below is the formal definition.
181
Lecture 23
Definition 23.4: A matrix A is called diagonalizable if it is similar to a diagonal matrix
D. In other words, if there exists an invertible P such that
A = PDP−1.
How do we determine when a given matrix A is diagonalizable? Let us first determine what
conditions need to be met for a matrix A to be diagonalizable. Suppose then that A is diag-
1
v v2 · · ·
vn
onalizable. Then by Definition 23.4, there exists an invertible matrix P = and a diagonal matrix
D =
1
. .
. .
.
.
λ 0 . . . 0
0 λ2 . . . 0
. . .
0 0 . . . λn
such that A = PDP−1. Multiplying on the right both sides of the equation A = PDP−1
by the matrix P we obtain that
AP = PD.
Now
while on the other hand
PD = λ v
1 1
λ2v2 · · ·
AP = Av1 Av2 · · · Avn
λnvn .
Therefore, since it holds that AP = PD then
Av1 Av2 · · · Avn = λ1v1 λ2v2 · · · λnvn .
or if we compare columns we must have that
Avi = λivi.
Thus, the columns v1, v2, . . . , vn of P are eigenvectors of A and form a basis for Rn because P is invertible. In conclusion, if A is diagonalizable then Rn has a basis consisting of eigenvectors of A.
Suppose instead that {v1, v2, . . . , vn} is a basis of Rn consisting of eigenvectors of A. Let
λ1, λ2, . . . , λn be the eigenvalues of A associated to v1, v2, . . . , vn, respectively, and set
P = v1 v2 · · · vn .
Then P is invertible because {v1, v2, . . . , vn} are linearly independent. Let
D =
1
. .
. .
.
.
λ 0 . . . 0
0 λ2 . . . 0
. . .
0 0 . . . λn
.
182
Diagonalization
Now, since Avi = λivi we have that
AP = A v1 v2 · · ·
vn
= Av1 Av2 · · · Avn
= λ1v1 λ2v2 · · · λnvn .
n n
Therefore, AP = λ1v1 λ2v2 · · · λ v . On the other hand,
PD = v1 v2 · · · v
n
1
0
. .
. .
.
.
λ 0 . . . 0
λ2 . . . 0
. . .
0 0 . . . λn
1 1 n n
= λ v λ2v2 · · · λ v .
Therefore, AP = PD, and since P is invertible we have that
A = PDP−1.
Thus, if Rn has a basis of consisting of eigenvectors of A then A is diagonalizable. We have therefore proved the following theorem.
Theorem 23.5: A matrix A is diagonalizable if and only if there is a basis {v1, v2, . . . , vn}
of Rn consisting of eigenvectors of A.
The punchline with Theorem 23.5 is that the problem of diagonalization of a matrix A is equivalent to finding a basis of Rn consisting of eigenvectors of A. We will see in some of the examples below that it is not always possible to diagonalize a matrix.
23.3 Conditions for Diagonalization
We first consider the simplest case when we conclude that a given matrix is diagonalizable, namely, the case when all eigenvalues are distinct.
Theorem 23.6: Suppose that A ∈ Rn×n has n distinct eigenvalues λ1, λ2, . . . , λn. Then
A is diagonalizable.
Proof. Each eigenvalue λi produces an eigenvector vi. The set of eigenvectors {v1, v2, . . . , vn} are linearly independent because they correspond to distinct eigenvalues (Theorem 21.6). Therefore, {v1, v2, . . . , vn} is a basis of Rn consisting of eigenvectors of A and then by Theorem 23.5 we conclude that A is diagonalizable.
What if A does not have distinct eigenvalues? Can A still be diagonalizable? The following theorem completely answers this question.
183
Lecture 23
Theorem 23.7: A matrix A is diagonalizable if and only if the algebraic and geometric multiplicities of each eigenvalue are equal.
Proof. Let A be a n × n matrix and let λ1, λ2, . . . , λp denote the distinct eigenvalues of A. Suppose that k1, k2, . . . , kp are the algebraic multiplicities and g1, g2, . . . , gp are the geometric multiplicities of the eigenvalues, respectively. Suppose that the algebraic and geometric multiplicities of each eigenvalue are equal, that is, suppose that gi = ki for each i = 1, 2 . . . , p. Since k1 +k2 +· · ·+kp = n, then because gi = ki we must also have that g1 +g2 +· · ·+gp = n. Therefore, there exists n linearly eigenvectors of A and consequently A is diagonalizable. On the other hand, suppose that A is diagonalizable. Since the geometric multiplicity is at most the algebraic multiplicity, the only way that g1 + g2 + · · · + gp = n is if gi = ki, i.e., that the geometric and algebraic multiplicities are equal.
Example 23.8. Determine if A is diagonalizable. If yes, find a matrix P that diagonalizes
A.
A = 3 5 3
0 0 3
−4 −6 −7
Solution. The characteristic polynomial of A is
p(λ) = det(A − λI) = (λ − 2)(λ − 3)(λ + 1)
and therefore λ1 = 2, λ2 = 3, and λ3 = −1 are the eigenvalues of A. Since A has n = 3 distinct eigenvalues, then by Theorem 23.6 A is diagonalizable. Eigenvectors v1, v2, v3 corresponding to λ1, λ2, λ3 are found to be
−1 −1 −2
1
v =
0
2
1 , v = 0
1
3
, v = 1
0
P = 1 0 1
0 1 0
Therefore, a matrix that diagonalizes A is
−1 −2 −2
You can verify that
λ1
P 0
0
λ2
0 0 λ3
0
−1
0 P = A
The following example demonstrates that it is possible for a matrix to be diagonalizable even though the matrix does not have distinct eigenvalues.
184
Diagonalization
Example 23.9. Determine if A is diagonalizable. If yes, find a matrix P that diagonalizes
A.
2 0
A = 4 2 2
−2 0 1
0
Solution. The characteristic polynomial of A is
2
p(λ) = det(A − λI) = (λ − 1)(λ − 2)
and therefore λ1 = 1, λ2 = 2. An eigenvector corresponding to λ1 = 1 is
1
v = −
0
2
1
One finds that g2 = dim(Null(A − λ2I)) = 2, and two linearly independent eigenvectors for
λ2 are
2 3
{v , v } =
0 , 1
2 0
−1 0
Therefore, A is diagonalizable, and a matrix that diagonalizes A is
0 −1
P = v1 v2 v3 = −
2 0 1
1 2 0
0
You can verify that
λ1
0
P 0
0
0 0 λ3
−1
λ2 0 P = A
Example 23.10. Determine if A is diagonalizable. If yes, find a matrix P that diagonalizes
A.
4
A = −
4 −6 −3
2 3
3 3 1
Solution. The characteristic polynomial of A is
3 2 2
p(λ) = det(A − λI) = −λ − 3λ + 4 = −(λ − 1)(λ + 2)
and therefore the eigenvalues of A are λ1 = 1 and λ2 = −2. For λ2 = −2 one computes
2
A − λ I ~
1
0 0 1
1 1
0 0 0
We see that the dimension of the eigenspace of λ2 = −2 is g2 = 1, which is less than the
algebraic multiplicity k2 = 2. Therefore, from Theorem 23.7 we can conclude that it is not possible to construct a basis of eigenvectors of A, and therefore A is not diagonalizable.
185
Lecture 23
Example 23.11. Suppose that A has eigenvector v with corresponding eigenvalue λ. Show
λ
that if A is invertible then v is an eigenvector of A−1 with corresponding eigenvalue 1 .
Example 23.12. Suppose that A and B are n × n matrices such that AB = BA. Show that if v is an eigenvector of A with corresponding eigenvalue then v is also an eigenvector of B with corresponding eigenvalue λ.
After this lecture you should know the following:
186
Diagonalization
Lecture 24
187
Lecture 24
Diagonalization of Symmetric Matrices
24.1 Symmetric Matrices
Recall that a square matrix A is said to be symmetric if AT = A. As an example, here is
a 3 × 3 symmetric matrix:
1 −3 7
A = −
3 2 8 .
7 8 4
Symmetric matrices are ubiquitous in mathematics. For example, let f(x1, x2, . . . , xn) be a function having continuous second order partial derivatives. Then Clairaut’s Theorem from multivariable calculus says that
∂f ∂f
= .
∂xi∂xj ∂xj ∂xi
Therefore, the Hessian matrix of f is symmetric:
Hess(f) =
∂f
∂x1 ∂x1
∂f
∂x1∂x2
· · ·
∂f
∂x1∂xn
∂f
∂x2∂xn
∂f
∂x2 ∂x1
∂f
∂x2∂x2
· · ·
.
.
. .
∂f ∂f
∂xn∂x1 ∂xn∂x2
. . .
· · ·
.
.
∂f
∂xn∂xn
.
The Second Derivative Test of multivariable calculus then says that if P = (a1, a2, . . . , an) is a critical point of f, that is
∂f ∂f ∂f
1 2 n
(P ) = (P ) = · · · = (P ) = 0
∂x ∂x ∂x
then
188
Diagonalization of Symmetric Matrices
(iii) P is a saddle point of f if the matrix Hess(f) has negative and positive eigenvalues.
In general, the eigenvalues of a matrix with real entries can be complex numbers. For example, the matrix
A =
0 −1
1 0
has characteristic polynomial
p(λ) = λ2 + 1
the roots of which are clearly λ1 = √−1 = i and λ2 = −√−1 = −i. Thus, in general, a matrix whose entries are all real numbers may have complex eigenvalues. However, for symmetric matrices we have the following.
Theorem 24.1: If A is a symmetric matrix then all of its eigenvalues are real numbers. The proof is easy but we will omit it.
24.2 Eigenvectors of Symmetric Matrices
We proved earlier that if {v1, v2, . . . , vk} are eigenvectors of a matrix A corresponding to distinct eigenvalues λ1, λ2, . . . , λk then the set {v1, v2, . . . , vk} is linearly independent (The- orem 21.6). For symmetric matrices we can say even more as the next theorem states.
Theorem 24.2: Let A be a symmetric matrix. If v1 and v2 are eigenvectors of A
corresponding to distinct eigenvalues then v1 and v2 are orthogonal, that is, v1 • v2 = 0.
•
1 2
T
1
2
/
Proof. Recall that v v = v v . Let λ = λ
1 2
be the eigenvalues associated to v1 and v2.
Then
1
λ1vT v2 = (λ1v1)T v2
= (Av1)T v2
1
= vT AT v2
1
= vT Av2
1
= vT (λ2v2)
1
= λ2vT v2.
T
1 2
T
T
1 1 1
2 2 1 2 2 1 2
/
Therefore, λ v v = λ v v which implies that (λ − λ )v v = 0. But since (λ − λ ) = 0
1
then we must have vT v2 = 0, that is, v1 and v2 are orthogonal.
24.3 Symmetric Matrices are Diagonalizable
As we have seen, the main criteria for diagonalization is that for each eigenvalue the geometric and algebraic multiplicities are equal; not all matrices satisfy this condition and thus not
189
Lecture 24
all matrices are diagonalizable. As it turns out, any symmetric A is diagonalizable and moreover (and perhaps more importantly) there exists an orthogonal eigenvector matrix P that diagonalizes A. The full statement is below.
Theorem 24.3: If A is a symmetric matrix then A is diagonalizable. In fact, there is an orthonormal basis of Rn of eigenvectors {v1, v2, . . . , vn} of A. In other words, the matrix P = [v1 v2 · · · vn] is orthogonal, PT P = I, and A = PDPT .
The proof of the theorem is not hard but we will omit it. The punchline of Theorem 24.3 is that, for the case of a symmetric matrix, we will never encounter the situation where the geometric multiplicity is strictly less than the algebraic multiplicity. Moreover, we are guaranteed to find an orthogonal matrix that diagonalizes a given symmetric matrix.
Example 24.4. Find an orthogonal matrix P that diagonalizes the symmetric matrix
1 0 −1
A = 0 1 1 .
−1 1 2
Solution. The characteristic polynomial of A is
p(λ) = det(A − λI) = λ3 − 4λ2 + 3λ = λ(λ − 1)(λ − 3)
The eigenvalues of A are λ1 = 0, λ2 = 1 and λ3 = 3. Eigenvectors of A associated to
λ1, λ2, λ3 are
1 1 −1
1
u = −
2 3
1 , u = 1 , u = 1 .
1 0 2
As expected by Theorem 24.2, the eigenvectors u1, u2, u3 form an orthogonal set:
uT u2 = 0, uT u3 = 0, uT u3 = 0.
1 1 2
To find an orthogonal matrix P that diagonalizes A we must normalize the eigenvectors
1 2 3
u , u , u to obtain an orthonormal basis
1 2 3
T
1
1
{v , v , v }. To that end, first compute u u = 3,
T T
2 3
2 3 1
1
√
3
1
2
√
2
2
3
1 1
√
6
3
u u = 2, and u u = 6. Then let v = u , let v = u , and let v = u . Therefore,
an orthogonal matrix that diagonalizes A is
3
P = v1 v2 v =
√
3
1 1
√
2
—
1
√
6
—
1
√
3
1
√
2
1
√
6
1
√
3
0
2
√
6
You can easily verify that PT P = I, and that
0
0
A = P 0
0
0 0 3
1 0 P
T
190
Diagonalization of Symmetric Matrices
Example 24.5. Let A and B be n × n matrices. Show that if A is symmetric then the matrix C = BABT is also a symmetric matrix.
After this lecture you should know the following:
Lecture 25
The PageRank Algortihm
In this lecture, we will see how linear algebra is used in Google’s webpage ranking algorithm used in everyday Google searches.
25.1 Search Engine Retrieval Process
Search engines perform a two-stage process to retrieve search results1. In Stage 1, traditional text processing is used to find all relevant pages (e.g. keywords in title, body) and produces a content score. After Stage 1, there is a large amount of relevant pages. For example, the query “symmetric matrix ” results in about 3,830,000 pages (03/31/15). Or “homework help” results in 49,400,000 pages (03/31/15). How should the relevant pages be displayed? In Stage 2, the pages are sorted and displayed based on a pre-computed ranking that is query-independent, this is the popularity score. The ranking is based on the hyperlinked or networked structure of the web, and the ranking is based on a popularity contest; if many pages link to page Pi then Pi must be an important page and should therefore have a high popularity score.
In January 1998, John Kleinberg from IBM (now a CS professor at Cornell) presented the HITS algorithm2 (e.g., www.teoma.com). At Stanford, doctoral students Sergey Brin and Larry Page were busy working on a similar project which they had begun in 1995. Below is the abstract of their paper3:
“In this paper, we present Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext. Google is designed to crawl and index the Web efficiently and produce much more satisfying search results than existing systems. The prototype with a full text and hyperlink database of at least 24 million pages is available at http://google.stanford.edu/ .”
1A.N. Langville and C.D. Meyer, Google’s PageRank and Beyond, Princeton University Press, 2006
2J. Kleinberg, Authoritative sources in a hyperlinked environment, Journal of ACM, 46, 1999, 9th ACM- SIAM Symposium on Discrete Algorithms
3S. Brin and L. Page, The anatomy of a large-scale hypertextual Web search engine, Computer Networks and ISDN Systems, 33:107-117, 1998
192
The PageRank Algortihm
In both models, the web is defined as a directed graph, where the nodes represent webpages and the directed arcs represent hyperlinks, see Figure 25.1.
1
3
2
4
Figure 25.1: A tiny web represented as a directed graph.
25.2 A Description of the PageRank Algorithm
In the PageRank algorithm, each inlink is viewed as a recommendation (or vote). In general, pages with many inlinks are more important than pages with few inlinks. However, the quality of the inlink (vote) is important. The vote of each page should be divided by the total number of recommendations made by the page. The PageRank of page i, denoted xi, is the sum of all the weighted PageRanks of all the pages pointing to i:
i
x =
Σ
x
j
j→i |Nj |
where
Example 25.1. Find the PageRank of each page for the network in Figure 25.1.
From the previous example, we see that the PageRank of each page can be found by solving an eigenvalue/eigenvector problem. However, when dealing with large networks such as the internet, the size of the problem is in the billions (8.1 billion in 2006) and directly solving the equations is not possible. Instead, an iterative method called the power method
4 4 4 4
is used. One starts with an initial guess, say x0 = (1 , 1 , 1 , 1 ). Then one updates the guess
by computing
x1 = Hx0.
In other words, we have a discrete dynamical system
xk+1 = Hxk.
A natural question is under what conditions will the the limiting value of the sequence
lim xk = lim (Hkx0) = q
k→∞ k→∞
193
k→∞
Lecture 25
converge to an equilibrium of H? Also, if lim xk exists, will it be a positive vector? And
lastly, can x0 /= 0 be chosen arbitrarily?
To see what situations may occur, consider the
0
1 1
5 5
network displayed in Figure 25.2. Starting with x = ( , . . . , ) we obtain that for k ≥ 39,
the vectors xk = Hkx0 cycle between (0, 0, 0, 0.28, 0.40) and (0, 0, 0, 0.40, 0.28). Therefore, the sequence x0, x1, x2, . . . does not converge. The reason for this is that nodes 4 and 5 form a cycle.
1
4
5
2 3
0 0
H = 0
1
3
0 0
0 0
1
2
0 0
1
3
0
0 0
0
1 1 0 1
3 2
0 0 0 1 0
Figure 25.2: Cycles present in the network
Now consider the network displayed in Figure 25.3. If we remove the cycle we are still left with a dangling node, namely node 1 (e.g. pdf file, image file). Starting with x0 = (1 , . . . , 1 ) results in
5 5
lim xk = 0.
k→∞
Therefore, in this case the sequence x0, x1, x2, . . . converges to a non-positive vector, which for the purposes of ranking pages would be an undesirable situation.
1
4
5
2 3
H = 0
0
1
3
0
0 0
0 0
0
1 1
2 2
0
1
3
0 0 0
1 1
3 2
0 0 0
1
2
0
0 1
Figure 25.3: Dangling node present in the network
To avoid the presence of dangling nodes and cycles, Brin and Page used the notion of a random surfer to adjust H. To deal with a dangling node, Brin and Page replaced
n n n n
the associated zero-column with the vector 1 1 = ( 1 , 1 , . . . , 1 ). The justification for this
adjustment is that if a random surfer reaches a dangling node, the surfer will “teleport” to any page in the web with equal probability. The new updated hyperlink matrix H∗ may still not have the desired properties. To deal with cycles, a surfer may abandon the hyperlink structure of the web by ocassionally moving to a random page by typing its address in the
194
The PageRank Algortihm
browser. With these adjustments, a random surfer now spends only a proportion of his time using the hyperlink structure of the web to visit pages. Hence, let 0 < α < 1 be the proportion of time the random surfer uses the hyperlink structure. Then the transition matrix is
∗ 1
n
G = αH + (1 − α) J.
The matrix G goes by the name of the Google matrix, and it is reported that Google uses α = 0.85 (here J is the all ones matrix). The Google matrix G is now a primitive and stochastic matrix. Stochastic means that all its columns are probability vectors, i.e., non- negative vectors whose components sum to 1. Primitive means that there exists k ≥ 1 such that Gk has all positive entries (k = 1 in our case). With these definitions, we now have the following theorem.
Theorem 25.2: If G is a primitive stochastic matrix then:
Proof. We will prove a special case4. Assume for simplicity that G is positive (this is the case of the Google Matrix). If x = Gx, and x has mixed signs, then
i
|x | =
n
Σ
j=1
ij j
n
Σ
G x < Gij |xj |.
j=1
Then
n
Σ
|x | <
n n
Σ Σ
Σ
i ij j j
G |x | = |x |
i=1 i=1 j=1 j=1
which is a contradiction. Therefore, all the eigenvectors in the λ1 = 1 eigenspace are either negative or positive. One then shows that the eigenspace corresponding to λ1 = 1 is 1- dimensional. This proves that there is a unique probability vector q such that
q = Gq.
4K. Bryan, T. Leise, The $25,000,000,000 Eigenvector: The Linear Algebra Behind Google, SIAM Review, 48(3), 569-581
195
Lecture 25
Let λ1, λ2, . . . , λn be the eigenvalues of G. We know that λ1 = 1 is a dominant eigenvalue:
|λ1| > |λj |, j = 2, 3, . . . , n.
Let q0 be a probability vector and let q be as above, and let v2, . . . , vn be the remaining eigenvectors of G. Then q0 = q + c2v2 + · · · + cnvn and therefore
Gkq0 = Gk(q + c2v2 + · · · + cnvn)
= Gkq + c2Gkv2 + · · · + cnGkvn
2
2 2
k k
n
n n
= q + c λ v + · · · + c λ v .
From this we see that
lim Gkq0 = q.
k→∞
25.3 Computation of the PageRank Vector
The Google matrix G is completely dense, which is computationally undesirable. nately,
Fortu-
n
G = αH + (1 − α) ee
∗ 1 T
1 T
1
n n
= α(H + 11 ) + (1 − α) 11
T
n
= αH + (αa + (1 − α)1) 1
1 T
and H is very sparse and requires minimal storage. A vector-matrix multiplication generally requires O(n2) computation (n ≈ 8, 000, 000, 000 in 2006). Estimates show that the average webpage has about 10 outlinks, so H has about 10n non-zero entries. This means that multiplication with H reduces to O(n) computation. Aside from being very simple, the power method is a matrix-free method, i.e., no manipulation of the matrix H is done. Brin and Page, and others, have confirmed that only 50-100 iterations are needed for a satisfactory approximation of the PageRank vector q for the web.
After this lecture you should know the following:
196
The PageRank Algortihm
Lecture 26
197
Lecture 26
Discrete Dynamical Systems
26.1 Discrete Dynamical Systems
Many interesting problems in engineering, science, and mathematics can be studied within the framework of discrete dynamical systems. Dynamical systems are used to model systems that change over time. The state of the system (economic, ecologic, engineering, etc.) is measured at discrete time intervals producing a sequence of vectors x0, x1, x2, . . .. The relationship between the vector xk and the next vector xk+1 is what constitutes a model.
Definition 26.1: A linear discrete dynamical system on Rn is an infinite sequence
{x0, x1, x2, . . .} of vectors in Rn and a matrix A such that
xk+1 = Axk.
The vectors xk are called the state of the dynamical system and x0 is the initial condition of the system. Once the initial condition x0 is fixed, the remaining state vectors x1, x2, . . . , can be found by iterating the equation xk+1 = Axk.
26.2 Population Model
Consider the dynamic system consisting of the population movement between a city and its suburbs. Let x ∈ R2 be the state population vector whose first component is the population of the city and the second component is the population of the suburbs:
c
s
x = .
For simplicity, we assume that c + s = 1, i.e., c and s are population percentages of the total population. Suppose that in the year 1900, the city population was c0 and the suburban population was s0. Suppose it is known that after each year 5% of the city’s population
198
Discrete Dynamical Systems
moves to the suburbs and that 3% of the suburban population moves to the city. Hence, the population in the city in year 1901 is
c1 = 0.95c0 + 0.03s0,
while the population in the suburbs in year 1901 is
s1 = 0.05c0 + 0.97s0.
The equations
c1 = 0.95c0 + 0.03s0
s1 = 0.05c0 + 0.97s0
can be written in matrix form as
1
c 0.95
0.03 c
0
" # " # " #
= .
s1 0.05 0.97 s0
Performing the same analysis for the next year, the population in 1902 is
c 0.95 0.03 c
2 1
" # " # " #
= .
s2 0.05 0.97 s1
Hence, the population movement is a linear dynamical system with matrix and state vector
0.05 0.97
A = , xk =
0.95 0.03 c
k
" # " #
sk
.
Suppose that the initial population state vector is
x0 =
0.70
" #
0.30
.
Then,
x1 = Ax0 =
"
0.95 0.03 0.70
0.05 0.97 0.30
# " # "
=
0.674
0.326
#
.
Then,
"
# "
0.95 0.03 0.674
0.05 0.97 0.326
x2 = Ax1 = =
# "
0.650
0.349
#
.
In a similar fashion, one can compute that up to 3 decimal places:
0.375
x500 = x1000 =
0.375
" # " #
, .
0.625 0.625
It seems as though the population distribution converges to a steady state or equilibrium. We predict that in the year 2400, 38% of the total population will live in the city and 62% in the suburbs.
Our computations in the population model indicate that the population distribution is reaching a sort of steady state or equilibrium, which we now define.
199
Lecture 26
Definition 26.2: Let xk+1 = Axk be a discrete dynamical system. An equilibrium
state for A is a vector q such that Aq = q.
Hence, if q is an equilibrium for A and the initial condition is x0 = q then x1 = Ax0 = x0, and x2 = Ax1 = x0, and iteratively we have that xk = x0 = q for all k. Thus, if the system starts at the equilibrium q then it remains at q for all time.
How do we find equilibrium states? If q is an equilibrium for A then from Aq = q we have that
Aq − q = 0
and therefore
(A − I)q = 0.
Therefore, q is an equilibrium for A if and only if q is in the nullspace of the matrix A − I:
q ∈ Null(A − I).
Example 26.3. Find the equilibrium states of the matrix from the population model
A =
"0.95 0.03#
0.05 0.97
.
Does the initial condition of the population x0 change the long term behavior of the discrete dynamical system? We will know the answer once we perform an eigenvalue analysis on A (Lecture 22). As a preview, we will use the fact that
xk = Akx0
and then write x0 in an appropriate basis that reveals how A acts on x0. To see how the last equation was obtained, notice that
x1 = Ax0
and therefore
x2 = Ax1 = A(Ax0) = A2x0
and therefore
x3 = Ax2 = A(A2x0) = A3x0
etc.
26.3 Stability of Discrete Dynamical Systems
We first formally define the notion of stability of a discrete dynamical system.
200
Discrete Dynamical Systems
Definition 26.4: Consider the discrete dynamical system xk+1 = Axk where A ∈ Rn×n. The origin 0 ∈ Rn is said to be asymptotically stable if for any initial condition x0 ∈ Rn of the dynamical system we have
lim xk = lim Akx0 = 0.
k→∞ k→∞
The following theorem characterizes when a discrete linear dynamical system is asymptoti- cally stable.
Theorem 26.5: Let λ1, . . . , λn be the eigenvalues of A. If |λj | < 1 for all j = 1, 2, . . . , n
then the origin 0 is asymptotically stable for xk+1 = Axk.
Solution. For simplicity, we suppose that A is diagonalizable. Let {v1, . . . , vn} be a basis of eigenvectors of A with eigenvalues λ1, . . . , λn respectively. Then, for any vector x0 ∈ Rn, there exists constants c1, . . . , cn such that
x0 = c1v1 + · · · + cnvn.
Now, for any integer k ≥ 1 we have that.
i
Akvi = λkvi
Then
xk = Akx0
= Ak(c1v1 + · · · + cnvn)
= c1Akv1 + · · · + cnAkvn
1
1 1
k k
n
n n
= c λ v + · · · + c λ v .
i
k→∞
k
i
Since |λ | < 1 we have that lim λ = 0. Therefore,
k→∞ k→∞
k
1
k 1 1
k
n
n n
lim x = lim (c λ v + · · · + c λ v )
k
1
1 1
n
k→∞ k→∞
k
n
= c lim λ v + · · · + c lim λ v
n
= 0v1 + · · · + 0vn
= 0.
This completes the proof.
201
Lecture 26
As an example of an asymptotically stable dynamical system, consider the 2D system
xk+1 =
1.1 −0.4
0.15 0.6
x.
The eigenvalues of A =
1.1 −0.4
0.15 0.6
are λ1 = 0.8 and λ2 = 0.9. Hence, by Theorem 26.5,
for any initial condition x0, the sequence {x0, x1, x2, . . . , } converges to the origin in R2. In
Figure 26.1, we plot four different state seq
0 1 2
uences {x , x , x , . . . , } corresponding to the four
7
−7
7
−7
distinct initial conditions x0 = 3 , x0 = 3 , x0 = −3 , and x0 = −3 . As expected,
all trajectories converge to the origin.
Figure 26.1: A 2D asymptotically stable linear system
After this lecture you should know the following: