CHAPTER 5�INNER PRODUCT SPACES

Elementary Linear Algebra

R. Larson (8 Edition)

5.1 Length and Dot Product in Rⁿ

5.2 Inner Product Spaces

5.3 Orthonormal Bases: Gram-Schmidt Process

5.4 Mathematical Models and Least Square Analysis

5.5 Applications of Inner Product Space

​

​

投影片設計製作者

淡江大學 電機系 翁慶昌 教授

CH 5 Linear Algebra Applied

​

​

​

​

Electric/Magnetic Flux (p.240) Heart Rhythm Analysis (p.255)

Work (p.248)

​

​

​

​

​

Revenue (p.266) Torque (p.277)

2/101

5.1 Length and Dot Product in Rⁿ

• Length:

The length of a vector in Rⁿ is given by

• Notes: The length of a vector is also called its norm.

Elementary Linear Algebra: Section 5.1, p.232

3/101

• Ex 1:

(a) In R⁵, the length of is given by

​

​

(b) In R³ the length of is given by

(v is a unit vector)

Elementary Linear Algebra: Section 5.1, p.232

4/101

• A standard unit vector in Rⁿ:

u and v have the same direction

u and v have the opposite direction

• Notes: (Two nonzero vectors are parallel)

Elementary Linear Algebra: Section 5.1, p.232

5/101

• Thm 5.1: (Length of a scalar multiple)

Let v be a vector in Rⁿ and c be a scalar. Then

Elementary Linear Algebra: Section 5.1, p.233

6/101

• Thm 5.2: (Unit vector in the direction of v)

If v is a nonzero vector in Rⁿ, then the vector

​

has length 1 and has the same direction as v. This vector u is called the unit vector in the direction of v.

Elementary Linear Algebra: Section 5.1, p.233

7/101

• Notes:

(1) The vector is called the unit vector in the direction of v.

​

(2) The process of finding the unit vector in the direction of v

is called normalizing the vector v.

Elementary Linear Algebra: Section 5.1, p.233

8/101

• Ex 2: (Finding a unit vector)

Find the unit vector in the direction of ,

and verify that this vector has length 1.

Elementary Linear Algebra: Section 5.1, p.233

9/101

• Distance between two vectors:

The distance between two vectors u and v in Rⁿ is

Elementary Linear Algebra: Section 5.1, p.234

10/101

• Ex 3: (Finding the distance between two vectors)

The distance between u = (0, 2, 2) and v = (2, 0, 1) is

Elementary Linear Algebra: Section 5.1, p.234

11/101

• Dot product in Rⁿ:

The dot product of and

is the scalar quantity

Elementary Linear Algebra: Section 5.1, p.235

12/101

• Thm 5.3: (Properties of the dot product)

If u, v, and w are vectors in Rⁿ and c is a scalar,

then the following properties are true.

(1)

(2)

(3)

(4)

(5) , and if and only if

Elementary Linear Algebra: Section 5.1, p.235

13/101

• Euclidean n-space:

Rⁿ was defined to be the set of all order n-tuples of real numbers. When Rⁿ is combined with the standard operations of vector addition, scalar multiplication, vector length, and the dot product, the resulting vector space is called Euclidean n-space.

Elementary Linear Algebra: Section 5.1, p.235

14/101

Sol:

• Ex 5: (Finding dot products)

• (b) (c) (d) (e)

Elementary Linear Algebra: Section 5.1, p.236

15/101

• Ex 6: (Using the properties of the dot product)

Given

​

Sol:

Find

Elementary Linear Algebra: Section 5.1, p.236

16/101

• Ex 7: (An example of the Cauchy - Schwarz inequality)

Verify the Cauchy - Schwarz inequality for u=(1, -1, 3)

and v=(2, 0, -1)

• Thm 5.4: (The Cauchy - Schwarz inequality)

If u and v are vectors in Rⁿ, then

( denotes the absolute value of )

Sol:

Elementary Linear Algebra: Section 5.1, p.237

17/101

• The angle between two vectors in Rⁿ:

​

• Note:

The angle between the zero vector and another vector is not defined.

Opposite

direction

Same

direction

Elementary Linear Algebra: Section 5.1, p.238

18/101

• Ex 8: (Finding the angle between two vectors)

Sol:

u and v have opposite directions.

Elementary Linear Algebra: Section 5.1, p.238

19/101

• Orthogonal vectors:

Two vectors u and v in Rⁿ are orthogonal if

• Note:

The vector 0 is said to be orthogonal to every vector.

Elementary Linear Algebra: Section 5.1, p.238

20/101

• Ex 10: (Finding orthogonal vectors)

Determine all vectors in Rⁿ that are orthogonal to u=(4, 2).

Let

Sol:

Elementary Linear Algebra: Section 5.1, p.238

21/101

• Thm 5.5: (The triangle inequality)

If u and v are vectors in Rⁿ, then

Pf:

• Note:

Equality occurs in the triangle inequality if and only if

the vectors u and v have the same direction.

Elementary Linear Algebra: Section 5.1, p.239

22/101

• Thm 5.6: (The Pythagorean theorem)

If u and v are vectors in Rⁿ, then u and v are orthogonal

if and only if

Elementary Linear Algebra: Section 5.1, p.239

23/101

• Dot product and matrix multiplication:

(A vector in Rⁿ

is represented as an n×1 column matrix)

Elementary Linear Algebra: Section 5.1, p.240

24/101

Key Learning in Section 5.1

• Find the length of a vector and find a unit vector.
• Find the distance between two vectors.
• Find a dot product and the angle between two vectors, determine orthogonality, and verify the Cauchy-Schwarz Inequality, the triangle inequality, and the Pythagorean Theorem.
• Use a matrix product to represent a dot product.

​

25/101

Keywords in Section 5.1

• length: 長度
• norm: 範數
• unit vector: 單位向量
• standard unit vector : 標準單位向量
• normalizing: 單範化
• distance: 距離
• dot product: 點積
• Euclidean n-space: 歐基里德n維空間
• Cauchy-Schwarz inequality: 科西-舒瓦茲不等式
• angle: 夾角
• triangle inequality: 三角不等式
• Pythagorean theorem: 畢氏定理

26/101

5.2 Inner Product Spaces

• Inner product:

Let u, v, and w be vectors in a vector space V, and let c be any scalar. An inner product on V is a function that associates a real number <u, v> with each pair of vectors u and v and satisfies the following axioms.

(1)

(2)

(3)

(4) and if and only if

Elementary Linear Algebra: Section 5.2, p.243

27/101

• Note:

Elementary Linear Algebra: Section 5.2, Addition

28/101

• Ex 1: (The Euclidean inner product for Rⁿ)

Show that the dot product in Rⁿ satisfies the four axioms of an inner product.

Sol:

By Theorem 5.3, this dot product satisfies the required four axioms. Thus it is an inner product on Rⁿ.

Elementary Linear Algebra: Section 5.2, p.243

29/101

• Ex 2: (A different inner product for Rⁿ)

Show that the function defines an inner product on R², where and .

Sol:

Elementary Linear Algebra: Section 5.2, p.244

30/101

• Note: (An inner product on Rⁿ)

Elementary Linear Algebra: Section 5.2, p.244

31/101

• Ex 3: (A function that is not an inner product)

Show that the following function is not an inner product on R³.

Sol:

Let

Axiom 4 is not satisfied.

Thus this function is not an inner product on R³.

Elementary Linear Algebra: Section 5.2, p.244

32/101

• Thm 5.7: (Properties of inner products)

Let u, v, and w be vectors in an inner product space V, and let c be any real number.

(1)

(2)

(3)

• Norm (length) of u:

• Note:

Elementary Linear Algebra: Section 5.2, p.245

33/101

u and v are orthogonal if .

• Distance between u and v:

• Angle between two nonzero vectors u and v:

• Orthogonal:

Elementary Linear Algebra: Section 5.2, p.246

34/101

• Notes:

(1) If , then v is called a unit vector.

​

(2)

(the unit vector in the

direction of v)

not a unit vector

Elementary Linear Algebra: Section 5.2, p.246

35/101

• Ex 6: (Finding inner product)

is an inner product

Elementary Linear Algebra: Section 5.2, p.246

36/101

• Properties of norm:

(1)

(2) if and only if

(3)

• Properties of distance:

(1)

(2) if and only if

(3)

Elementary Linear Algebra: Section 5.2, p.247

37/101

• Thm 5.8：

Let u and v be vectors in an inner product space V.

(1) Cauchy-Schwarz inequality:

(2) Triangle inequality:

(3) Pythagorean theorem :

u and v are orthogonal if and only if

Theorem 5.5

Theorem 5.6

Theorem 5.4

Elementary Linear Algebra: Section 5.2, p.248

38/101

• Orthogonal projections in inner product spaces:

Let u and v be two vectors in an inner product space V, such that . Then the orthogonal projection of u onto v is given by

• Note:

If v is a init vector, then .

The formula for the orthogonal projection of u onto v takes the following simpler form.

Elementary Linear Algebra: Section 5.2, p.249

39/101

• Ex 10: (Finding an orthogonal projection in R³)

Use the Euclidean inner product in R³ to find the orthogonal projection of u=(6, 2, 4) onto v=(1, 2, 0).

Sol:

Elementary Linear Algebra: Section 5.2, p.249

40/101

• Thm 5.9: (Orthogonal projection and distance)

Let u and v be two vectors in an inner product space V, such that . Then

Elementary Linear Algebra: Section 5.2, p.260

41/101

Key Learning in Section 5.2

• Determine whether a function defines an inner product, and find the inner product of two vectors in Rⁿ, M_m,n, P_n and C[a, b].
• Find an orthogonal projection of a vector onto another vector in an inner product space.

42/101

Keywords in Section 5.2

• inner product: 內積
• inner product space: 內積空間
• norm: 範數
• distance: 距離
• angle: 夾角
• orthogonal: 正交
• unit vector: 單位向量
• normalizing: 單範化
• Cauchy – Schwarz inequality: 科西 - 舒瓦茲不等式
• triangle inequality: 三角不等式
• Pythagorean theorem: 畢氏定理
• orthogonal projection: 正交投影

43/101

5.3 Orthonormal Bases: Gram-Schmidt Process

• Orthogonal:

A set S of vectors in an inner product space V is called an orthogonal set if every pair of vectors in the set is orthogonal.

• Orthonormal:

An orthogonal set in which each vector is a unit vector is called orthonormal.

​

• Note:

If S is a basis, then it is called an orthogonal basis or an orthonormal basis.

Elementary Linear Algebra: Section 5.3, p.254

44/101

• Ex 1: (A nonstandard orthonormal basis for R³)

Show that the following set is an orthonormal basis.

​

Elementary Linear Algebra: Section 5.3, p.255

45/101

Show that each vector is of length 1.

Thus S is an orthonormal set.

Elementary Linear Algebra: Section 5.3, p.255

46/101

The standard basis is orthonormal.

• Ex 2: (An orthonormal basis for )

In , with the inner product

Elementary Linear Algebra: Section 5.3, p.255

47/101

Elementary Linear Algebra: Section 5.3, p.255

48/101

• Thm 5.10: (Orthogonal sets are linearly independent)

If is an orthogonal set of nonzero vectors in an inner product space V, then S is linearly independent.

Pf:

S is an orthogonal set of nonzero vectors

Elementary Linear Algebra: Section 5.3, p.257

49/101

• Corollary to Thm 5.10:

If V is an inner product space of dimension n, then any orthogonal set of n nonzero vectors is a basis for V.

Elementary Linear Algebra: Section 5.3, p.257

50/101

• Ex 4: (Using orthogonality to test for a basis)

Show that the following set is a basis for .

Sol:

​

​

: nonzero vectors

(by Corollary to Theorem 5.10)

Elementary Linear Algebra: Section 5.3, p.257

51/101

• Thm 5.11: (Coordinates relative to an orthonormal basis)

If is an orthonormal basis for an inner product space V, then the coordinate representation of a vector w with respect to B is

is orthonormal

(unique representation)

Pf:

is a basis for V

Elementary Linear Algebra: Section 5.3, p.258

52/101

• Note:

If is an orthonormal basis for V and ,

Then the corresponding coordinate matrix of w relative to B is

Elementary Linear Algebra: Section 5.3, p.258

53/101

• Ex 5: (Representing vectors relative to an orthonormal basis)

Find the coordinates of w = (5, -5, 2) relative to the following orthonormal basis for .

Elementary Linear Algebra: Section 5.3, p.258

54/101

• Gram-Schmidt orthonormalization process:

is a basis for an inner product space V

is an orthogonal basis.

is an orthonormal basis.

Elementary Linear Algebra: Section 5.3, p.259

55/101

Sol:

• Ex 7: (Applying the Gram-Schmidt orthonormalization process)

Apply the Gram-Schmidt process to the following basis.

Elementary Linear Algebra: Section 5.3, p.260

56/101

Orthogonal basis

Orthonormal basis

Elementary Linear Algebra: Section 5.3, p.260

57/101

• Ex 10: (Alternative form of Gram-Schmidt orthonormalization process)

Find an orthonormal basis for the solution space of the

homogeneous system of linear equations.

Sol:

Elementary Linear Algebra: Section 5.3, p.262

58/101

Thus one basis for the solution space is

(orthogonal basis)

(orthonormal basis)

Elementary Linear Algebra: Section 5.3, p.262

59/101

Key Learning in Section 5.3

• Show that a set of vectors is orthogonal and forms an orthonormal basis, and represent a vector relative to an orthonormal basis.
• Apply the Gram-Schmidt orthonormalization process.

60/101

Keywords in Section 5.3

• orthogonal set: 正交集合
• orthonormal set: 單範正交集合
• orthogonal basis: 正交基底
• orthonormal basis: 單範正交基底
• linear independent: 線性獨立
• Gram-Schmidt Process: Gram-Schmidt過程

61/101

5.4 Mathematical Models and Least Squares Analysis

• Orthogonal subspaces:

Elementary Linear Algebra: Section 5.4, p.266

• Ex 2: (Orthogonal subspaces)

62/101

Let W be a subspace of an inner product space V.

(a) A vector u in V is said to orthogonal to W,

if u is orthogonal to every vector in W.

(b) The set of all vectors in V that are orthogonal to every

vector in W is called the orthogonal complement of W.

• Orthogonal complement of W:

Elementary Linear Algebra: Section 5.4, p.266

63/101

• Notes:

• Ex:

Elementary Linear Algebra: Section 5.4, Addition

64/101

• Direct sum:

Let and be two subspaces of . If each vector can be uniquely written as a sum of a vector from and a vector from , , then is the direct sum of and , and you can write .

Elementary Linear Algebra: Section 5.4, pp.267-268

65/101

• Thm 5.14: (Projection onto a subspace)

If is an orthonormal basis for the subspace S of V, and , then

Elementary Linear Algebra: Section 5.4, p.268

Pf:

66/101

• Ex 5: (Projection onto a subspace)

Find the projection of the vector v onto the subspace W.

Sol:

an orthogonal basis for W

an orthonormal basis for W

Elementary Linear Algebra: Section 5.4, p.269

67/101

• Find by the other method:

Elementary Linear Algebra: Section 5.4, p.269

68/101

• Thm 5.15: (Orthogonal projection and distance)

Let W be a subspace of an inner product space V, and .

Then for all ,

( is the best approximation to v from W)

Elementary Linear Algebra: Section 5.4, p.269

69/101

Pf:

By the Pythagorean theorem

Elementary Linear Algebra: Section 5.4, p.269

70/101

• Notes:

(1) Among all the scalar multiples of a vector u, the

orthogonal projection of v onto u is the one that is

closest to v. (p.250 Thm 5.9)

(2) Among all the vectors in the subspace W, the vector

is the closest vector to v.

​

Elementary Linear Algebra: Section 5.4, p.269

71/101

• Thm 5.16: (Fundamental subspaces of a matrix)

If A is an m × n matrix, then

(1)

​

(2)

​

(3)

(4)

Elementary Linear Algebra: Section 5.4, p.270

72/101

• Ex 6: (Fundamental subspaces)

Find the four fundamental subspaces of the matrix.

(reduced row-echelon form)

Sol:

Elementary Linear Algebra: Section 5.4, p.270

73/101

• Check:

Elementary Linear Algebra: Section 5.4, p.270

74/101

• Ex 3 & Ex 4:

​

Let W is a subspace of R⁴ and .

(a) Find a basis for W

(b) Find a basis for the orthogonal complement of W.

Elementary Linear Algebra: Section 5.4, p.267

75/101

• Notes:

is a basis for W

Elementary Linear Algebra: Section 5.4, p.267

76/101

• Least squares problem:

(A system of linear equations)

(1) When the system is consistent, we can use the Gaussian elimination with back-substitution to solve for x

(2) When the system is inconsistent, how to find the “best possible” solution of the system. That is, the value of x for which the difference between Ax and b is small.

Elementary Linear Algebra: Section 5.4, p.271

77/101

• Notes:

• Least squares solution:

Given a system Ax = b of m linear equations in n unknowns, the least squares problem is to find a vector x in Rⁿ that minimizes with respect to the Euclidean inner product on Rⁿ. Such a vector is called a least squares solution of Ax = b.

Elementary Linear Algebra: Section 5.4, p.271

78/101

Elementary Linear Algebra: Section 5.4, p.271

79/101

• Note: (Ax = b is an inconsistent system)

The problem of finding the least squares solution of

is equal to he problem of finding an exact solution of the associated normal system .

Elementary Linear Algebra: Section 5.4, p.271

80/101

• Ex 7: (Solving the normal equations)

Find the least squares solution of the following system

​

​

(this system is inconsistent)

​

and find the orthogonal projection of b on the column space of A.

​

Elementary Linear Algebra: Section 5.4, p.271

81/101

Sol:

the associated normal system

Elementary Linear Algebra: Section 5.4, p.271

82/101

the least squares solution of Ax = b

the orthogonal projection of b on the column space of A

Elementary Linear Algebra: Section 5.4, p.271

83/101

Key Learning in Section 5.4

• Define the least squares problem.
• Find the orthogonal complement of a subspace and the projection of a vector onto a subspace.
• Find the four fundamental subspaces of a matrix.
• Solve a least squares problem.
• Use least squares for mathematical modeling.

84/101

Keywords in Section 5.4

• orthogonal to W: 正交於W
• orthogonal complement: 正交補集
• direct sum: 直和
• projection onto a subspace: 在子空間的投影
• fundamental subspaces: 基本子空間
• least squares problem: 最小平方問題
• normal equations: 一般方程式

85/101

5.5 Applications of Inner Product Spaces

• The cross product of two vectors in R³

A vector product that yields a vector in R³ is orthogonal to two vectors. This vector product is called the cross product, and it is most conveniently defined and calculated with vectors written in standard unit vector form

Elementary Linear Algebra: Section 5.5, p.277

86/101

• Cross product of two vectors in R³:

Let and be vectors in R³.

The cross product of u and v is the vector

Elementary Linear Algebra: Section 5.5, p.277

• Notes:

(1) The cross product is defined only for vectors in R³.

(2) The cross product of two vectors in R³ is orthogonal to two vectors.

(3) The cross product of two vectors in Rⁿ, n ≠ 3 is not defined here.

87/101

• Ex 1: (Finding the Cross Product of Two Vectors)

Sol:

Elementary Linear Algebra: Section 5.5, p.278

88/101

• Thm 5.17: (Algebraic Properties of the Cross Product)

If u, v, and w are vectors in R³ and c is a scalar, then the following properties are true.

Elementary Linear Algebra: Section 5.5, p.278

89/101

Relation between inner and cross products

90/101

Pf:

• Note:

The vectors u × v and v × u have equal lengths but opposite directions.

Elementary Linear Algebra: Section 5.5, p.278-279

91/101

• Thm 5.18: (Geometric Properties of the Cross Product)

If u and v are nonzero vectors in R³, then the following properties are true.

1. u × v is orthogonal to both u and v.

2. The angle θ between u and v is given by .

3. u and v are parallel if and only if .

4. The parallelogram having u and v as adjacent sides has an

area of .

Elementary Linear Algebra: Section 5.5, p.279

92/101

Pf:

• Notes:

(1) The three vectors u, v, and u × v form a right-handed system.

(2) The three vectors u, v, and v × u form a left-handed system.

Elementary Linear Algebra: Section 5.5, p.279

93/101

Ex 2: (Finding a Vector Orthogonal to Two Given Vectors)

Sol:

length

unit vector

Elementary Linear Algebra: Section 5.5, p.280

94/101

Ex 3: (Finding the Area of a Parallelogram)

Sol:

area

Elementary Linear Algebra: Section 5.5, p.280

95/101

Key Learning in Section 5.5

• Find the cross product of two vectors in R³.
• Find the linear or quadratic least squares approximation of a function.
• Find the nth-order Fourier approximation of a function.

96/101

Keywords in Section 5.5

• cross product: 外積
• parallelogram: 平行四邊形

​

​

​

​

97/101

• Electric/Magnetic Flux

​

Electrical engineers can use the dot product to calculate electric or magnetic flux, which is a measure of the strength of the electric or magnetic field penetrating a surface. Consider an arbitrarily shaped surface with an element of area dA, normal (perpendicular) vector dA, electric field vector E and magnetic field vector B. The electric flux Φ_e can be found using the surface integral Φ_e = ∫ E‧dA and the magnetic flux can be found using the surface integral Φ_e = ∫ B‧dA. It is interesting to note that for a given closed surface that surrounds an electrical charge, the net electric flux is proportional to the charge, but the net magnetic flux is zero. This is because electric fields initiate at positive charges and terminate at negative charges, but magnetic fields form closed loops, so they do not initiate or terminate at any point. This means that the magnetic field entering a closed surface must equal the magnetic field leaving the closed surface.

5.1 Linear Algebra Applied

Elementary Linear Algebra: Section 5.1, p.240

98/101

• Work

​

The concept of work is important to scientists and engineers for determining the energy needed to perform various jobs. If a constant force F acts at an angle with the line of motion of an object to move the object from point A to point B (see figure below), then the work done by the force is given by

​

​

​

where represents the directed line segment from A to B. The quantity is the length of the orthogonal projection of F onto Orthogonal projections are discussed on the next page.

​

5.2 Linear Algebra Applied

Elementary Linear Algebra: Section 5.2, p.248

99/101

• Heart Rhythm Analysis

​

Time-frequency analysis of irregular physiological signals, such as beat-to-beat cardiac rhythm variations (also known as heart rate variability or HRV), can be difficult. This is because the structure of a signal can include multiple periodic, nonperiodic, and pseudo-periodic components. Researchers have proposed and validated a simplified HRV analysis method called orthonormal-basis partitioning and time-frequency representation (OPTR). This method can detect both abrupt and slow changes in the HRV signal’s structure, divide a nonstationary HRV signal into segments that are “less nonstationary,” and determine patterns in the HRV. The researchers found that although it had poor time resolution with signals that changed gradually, the OPTR method accurately represented multicomponent and abrupt changes in both real-life and simulated HRV signals.

5.3 Linear Algebra Applied

Elementary Linear Algebra: Section 5.3, p.255

100/101

• Revenues

​

The least squares problem has a wide variety of real-life applications. To illustrate, in Examples 9 and 10 and Exercises 39, 40, and 41, are all least squares analysis problems, and they involve such diverse subject matter as world population, astronomy, master’s degrees awarded, company revenues, and galloping speeds of animals. In each of these applications, you will be given a set of data and you are asked to come up with mathematical model(s) for the data. For example, in Exercise 40, you are given the annual revenues from 2008 through 2013 for General Dynamics Corporation. You are asked to find the least squares regression quadratic and cubic polynomials for the data, to predict the revenue for the year 2018, and to decide which of the models appears to be more accurate for predicting future revenues.

5.4 Linear Algebra Applied

Elementary Linear Algebra: Section 5.4, p.266

101/101

• Torque

​

In physics, the cross product can be used to measure torque—the moment M of a force F about a point A as shown in the figure below. When the point of application of the force is B, the moment of F about A is given by

​

​

where represents the vector whose initial point is A and whose terminal point is B. The magnitude of the moment M measures the tendency of to rotate counterclockwise about an axis directed along the vector M.

​

5.5 Linear Algebra Applied

Elementary Linear Algebra: Section 5.5, p.277

102/101