Chapter 7 Eigenvalues and Eigenvectors 7.1 Eigenvalues and Eigenvectors 7.2 Diagonalization 7.3 Symmetric Matrices and Orthogonal Diagonalization 7.4 Application of Eigenvalues and Eigenvectors 7.5 Principal Component Analysis 7. 1 7. 2 7.1 Eigenvalues and Eigenvectors n Eigenvalue problem ( 特徵值問題 ) ( one of the most important problems in the linear algebra): If A is an n ́ n matrix, do there exist nonzero vectors x in R n such that A x is a scalar multiple of x ? n Eigenvalue ( 特徵值 ) and Eigenvector ( 特徵向量 ): A : an n ́ n matrix l : a scalar (could be zero ) x : a nonzero vector in R n A l = x x Eigenvalue Eigenvector ※ Geometric Interpretation (The term eigenvalue is from the German word Eigenwert , meaning “proper value”) x A l x = x x y 7. 3 n Ex 1: Verifying eigenvalues and eigenvectors ú û ù ê ë é - = 1 0 0 2 A 1 1 0 é ù = ê ú ë û x 1 1 2 0 1 2 1 2 2 0 1 0 0 0 A é ù é ù é ù é ù = = = = ê ú ê ú ê ú ê ú - ë û ë û ë û ë û x x Eigenvalue 2 2 2 0 0 0 0 1 ( 1) 0 1 1 1 1 A é ù é ù é ù é ù = = = - = - ê ú ê ú ê ú ê ú - - ë û ë û ë û ë û x x Eigenvalue Eigenvector Eigenvector 2 0 1 é ù = ê ú ë û x ※ In fact, for each eigenvalue , it has infinitely many eigenvectors. For l = 2, [3 0] T or [5 0] T are both corresponding eigenvectors. Moreover, ([3 0] + [5 0]) T is still an eigenvector. The proof is in Thm . 7.1. 7. 4 n Thm. 7.1: The eigenspace corresponding to l of matrix A If A is an n ́ n matrix with an eigenvalue l , then the set of all eigenvectors of l together with the zero vector is a subspace of R n . This subspace is called the eigenspace ( 特徵空間 ) of l Pf: x 1 and x 2 are eigenvectors corresponding to l 1 1 2 2 (i.e., , ) A A l l = = x x x x 1 2 1 2 1 2 1 2 1 2 (1) ( ) ( ) (i.e., is also an eigenvector cor responding to ) A A A λ l l l + = + = + = + + x x x x x x x x x x 1 1 1 1 1 (2) ( ) ( ) ( ) ( ) (i.e., is also an eigenvector cor responding to ) A c c A c c c l l l = = = x x x x x Since this set is closed under vector addition and scalar multiplication, this set is a subspace of R n according to Theorem 4.5 7. 5 n Ex 3: Examples of eigenspaces on the xy - plane For the matrix A as follows, the corresponding eigenvalues are l 1 = – 1 and l 2 = 1: ú û ù ê ë é - = 1 0 0 1 A Sol: 0 1 0 0 0 0 1 0 1 A y y y y - é ù é ù é ù é ù é ù = = = ê ú ê ú ê ú ê ú ê ú ë û ë û ë û ë û ë û For the eigenvalue l 1 = – 1, corresponding vectors are any vectors on the x - axis 1 0 1 0 0 1 0 0 0 x x x x A - - é ù é ù é ù é ù é ù = = = - ê ú ê ú ê ú ê ú ê ú ë û ë û ë û ë û ë û For the eigenvalue l 2 = 1, corresponding vectors are any vectors on the y - axis ※ Thus, the eigenspace corresponding to l = – 1 is the x - axis, which is a subspace of R 2 ※ Thus, the eigenspace corresponding to l = 1 is the y - axis, which is a subspace of R 2 7. 6 ※ Geometrically speaking, multiplying a vector ( x , y ) in R 2 by the matrix A corresponds to a reflection to the y - axis, i.e., left multiplying A to v can transform v to another vector in the same vector space 0 0 0 0 0 1 1 0 x x x A A A A A y y y x x y y æ ö é ù é ù é ù é ù é ù = = + = + ç ÷ ê ú ê ú ê ú ê ú ê ú ë û ë û ë û ë û ë û è ø - é ù é ù é ù = - + = ê ú ê ú ê ú ë û ë û ë û v 7. 7 (1) An eigenvalue of A is a scalar l such that n Thm. 7.2 : Finding eigenvalues and eigenvectors of a matrix A Î M n ́ n det( ) 0 I A l - = (2) The eigenvectors of A corresponding to l are the nonzero solutions of n Characteristic polynomial ( 特徵多項式 ) of A Î M n ́ n : 1 1 1 0 det( ) ( ) n n n I A I A c c c l l l l l - - - = - = + + + + ! n Characteristic equation ( 特徵方程式 ) of A : det( ) 0 I A l - = ( ) I A l - = x 0 Let A be an n ́ n matrix. has nonzero solutions for x iff ( ) I A l - = x 0 det( ) 0 I A l - = n Note: follwing the definition of the eigenvalue problem (homogeneous system) ( ) A A I I A l l l = Þ = Þ - = x x x x x 0 (The above iff results comes from the equivalent conditions on Slide 4.101) 7. 8 n Ex 4: Finding eigenvalues and eigenvectors ú û ù ê ë é - - = 5 1 12 2 A Sol: Characteristic equation: 2 2 12 det( ) 1 5 3 2 ( 1)( 2) 0 I A l l l l l l l - - = - + = + + = + + = Eigenvalue: 2 , 1 2 1 - = - = l l 2 , 1 - - = Þ l 7. 9 2 (2) 2 l = - 1 2 2 G.-J. E. 1 2 4 12 0 ( ) 1 3 0 4 12 1 3 1 3 0 0 3 3 , 0 1 x I A x x s s s x s l - é ù é ù é ù Þ - = = ê ú ê ú ê ú - ë û ë û ë û - - é ù é ù Þ ¾¾¾ ® ê ú ê ú - ë û ë û é ù é ù é ù Þ = = ¹ ê ú ê ú ê ú ë û ë û ë û x 1 (1) 1 l = - 1 1 2 G.-J. E. 1 2 3 12 0 ( ) 1 4 0 3 12 1 4 1 4 0 0 4 4 , 0 1 x I A x x t t t x t l - é ù é ù é ù Þ - = = ê ú ê ú ê ú - ë û ë û ë û - - é ù é ù Þ ¾¾¾ ® ê ú ê ú - ë û ë û é ù é ù é ù Þ = = ¹ ê ú ê ú ê ú ë û ë û ë û x 7. 10 ú ú û ù ê ê ë é = 2 0 0 0 2 0 0 1 2 A Sol: Characteristic equation: 3 2 1 0 0 2 0 ( 2) 0 0 0 2 I A l l l l l - - - = - = - = - Eigenvalue: 2 = l n Ex 5: Finding eigenvalues and eigenvectors Find the eigenvalues and corresponding eigenvectors for the matrix A . What is the dimension of the eigenspace of each eigenvalue? 7. 11 The eigenspace of λ = 2: 1 2 3 0 1 0 0 ( ) 0 0 0 0 0 0 0 0 x I A x x l - é ù é ù é ù ê ú ê ú ê ú - = = ê ú ê ú ê ú ê ú ê ú ê ú ë û ë û ë û x 0 , , 1 0 0 0 0 1 0 3 2 1 ¹ ú ú û ù ê ê ë é + ú ú û ù ê ê ë é = ú ú û ù ê ê ë é = ú ú û ù ê ê ë é t s t s t s x x x 1 0 0 0 , : the eigenspace of corresponding to 2 0 1 s t s t R A l ì ü é ù é ù ï ï ê ú ê ú + Î = í ý ê ú ê ú ï ï ê ú ê ú ë û ë û î þ Thus, the dimension of its eigenspace is 2 7. 12 n Notes: (1) If an eigenvalue l 1 occurs as a multiple root ( k times) for the characteristic polynominal , then l 1 has multiplicity ( 重數 ) k (2) The multiplicity of an eigenvalue is greater than or equal to the dimension of its eigenspace (In Ex. 5, k is 3 and the dimension of its eigenspace is 2) 7. 13 n Ex 6 ： Find the eigenvalues of the matrix A and find a basis for each of the corresponding eigenspaces ú ú ú ú û ù ê ê ê ê ë é - = 3 0 0 1 0 2 0 1 10 5 1 0 0 0 0 1 A Sol: Characteristic equation: 2 1 0 0 0 0 1 5 10 1 0 2 0 1 0 0 3 ( 1) ( 2)( 3) 0 I A l l l l l l l l - - - - = - - - - = - - - = Eigenvalues: 3 , 2 , 1 3 2 1 = = = l l l ※ According to the note on the previous slide, the dimension of the eigenspace of λ 1 = 1 is at most to be 2 ※ For λ 2 = 2 and λ 3 = 3, the demensions of their eigenspaces are at most to be 1 7. 14 1 (1) 1 l = 1 2 1 3 4 0 0 0 0 0 0 0 5 10 0 ( ) 1 0 1 0 0 1 0 0 2 0 x x I A x x l é ù é ù é ù ê ú ê ú ê ú - ê ú ê ú ê ú Þ - = = ê ú ê ú ê ú - - ê ú ê ú ê ú - - ë û ë û ë û x 1 G.-J.E. 2 3 4 2 0 2 1 0 , , 0 2 0 2 0 1 x t x s s t s t x t x t - - é ù é ù é ù é ù ê ú ê ú ê ú ê ú ê ú ê ú ê ú ê ú Þ = = + ¹ ê ú ê ú ê ú ê ú ê ú ê ú ê ú ê ú ë û ë û ë û ë û 1 2 0 2 , 0 0 1 0 ï ï þ ï ï ý ü ï ï î ï ï í ì ú ú ú ú û ù ê ê ê ê ë é - ú ú ú ú û ù ê ê ê ê ë é Þ is a basis for the eigenspace corresponding to 1 1 l = ※The dimension of the eigenspace of λ 1 = 1 is 2 7. 15 2 (2) 2 l = 1 2 2 3 4 1 0 0 0 0 0 1 5 10 0 ( ) 1 0 0 0 0 1 0 0 1 0 x x I A x x l é ù é ù é ù ê ú ê ú ê ú - ê ú ê ú ê ú Þ - = = ê ú ê ú ê ú - ê ú ê ú ê ú - - ë û ë û ë û x 1 G.-J.E. 2 3 4 0 0 5 5 , 0 1 0 0 x x t t t x t x é ù é ù é ù ê ú ê ú ê ú ê ú ê ú ê ú Þ = = ¹ ê ú ê ú ê ú ê ú ê ú ê ú ë û ë û ë û 0 1 5 0 ï ï þ ï ï ý ü ï ï î ï ï í ì ú ú ú ú û ù ê ê ê ê ë é Þ is a basis for the eigenspace corresponding to 2 2 l = ※The dimension of the eigenspace of λ 2 = 2 is 1 7. 16 3 (3) 3 l = 1 2 3 3 4 2 0 0 0 0 0 2 5 10 0 ( ) 1 0 1 0 0 1 0 0 0 0 x x I A x x l é ù é ù é ù ê ú ê ú ê ú - ê ú ê ú ê ú Þ - = = ê ú ê ú ê ú - ê ú ê ú ê ú - ë û ë û ë û x 1 G.-J.E. 2 3 4 0 0 5 5 , 0 0 0 1 x x t t t x x t é ù é ù é ù ê ú ê ú ê ú - - ê ú ê ú ê ú Þ = = ¹ ê ú ê ú ê ú ê ú ê ú ê ú ë û ë û ë û 1 0 5 0 ï ï þ ï ï ý ü ï ï î ï ï í ì ú ú ú ú û ù ê ê ê ê ë é - Þ is a basis for the eigenspace corresponding to 3 3 l = ※The dimension of the eigenspace of λ 3 = 3 is 1 7. 17 n Thm. 7.3: Eigenvalues for triangular matrices If A is an n ́ n triangular matrix, then its eigenvalues are the entries on its main diagonal n Ex 7: Finding eigenvalues for triangular and diagonal matrices 2 0 0 (a) 1 1 0 5 3 3 A é ù ê ú = - ê ú ê ú - ë û 1 0 0 0 0 0 2 0 0 0 (b) 0 0 0 0 0 0 0 0 4 0 0 0 0 0 3 A - é ù ê ú ê ú ê ú = ê ú - ê ú ê ú ë û Sol: 2 0 0 (a) 1 1 0 ( 2)( 1)( 3) 0 5 3 3 I A l l l l l l l - - = - = - - + = - - + 1 2 3 2, 1, 3 l l l Þ = = = - 1 2 3 4 5 (b) 1, 2, 0, 4, 3 l l l l l = - = = = - = ※According to Thm . 3.2, the determinant of a triangular matrix is the product of the entries on the main diagonal 7. 18 n Eigenvalues and eigenvectors of linear transformations: A number is called an eigenvalue of a l inear transformation : if there is a nonzero vector such tha t ( ) The vector is called an eigenvector of corresponding to , and the set of all T V V T T l l l ® = x x x x eigenvectors of (together with the zero vector) is called the eigenspace of l l ※ The definition of linear transformation functions should be introduced in Ch 6 ※ Here I briefly introduce the linear transformation and its some basic properties ※ The typical example of a linear transformation function is that each component of the resulting vector is the linear combination of the components in the input vector x n An example for a linear transformation T : R 3 → R 3 1 2 3 1 2 1 2 3 ( , , ) ( 3 ,3 , 2 ) T x x x x x x x x = + + - 7. 19 n Theorem: Standard matrix for a linear transformation Let : be a linear trtansformation such t hat n n T R R ® 11 12 1 21 22 2 1 2 1 2 ( ) , ( ) , , ( ) , n n n n n nn a a a a a a T T T a a a é ù é ù é ù ê ú ê ú ê ú ê ú ê ú ê ú = = = ê ú ê ú ê ú ê ú ê ú ê ú ë û ë û ë û e e e ! " " " 1 2 n where { , , , } is a standard basis for . Th en an matrix , whose -th column correspond to ( ), n i R n n A i T ́ e e e e ! satisfies that ( ) for every in is ca lled the standard matrix for ( ) n T A R A T T = x x x 的 標 準 矩陣 [ ] 11 12 1 21 22 2 1 2 1 2 ( ) ( ) ( ) , n n n n n nn a a a a a a A T T T a a a é ù ê ú ê ú = = ê ú ê ú ë û e e e ! ! ! " " # " ! 7. 20 n Consider the same linear transformation T ( x 1 , x 2 , x 3 ) = ( x 1 + 3 x 2 , 3 x 1 + x 2 , – 2 x 3 ) 1 2 3 1 1 0 3 0 0 ( ) ( 0 ) 3 , ( ) ( 1 ) 1 , ( ) ( 0 ) 0 0 0 0 0 1 2 T T T T T T é ù é ù é ù é ù é ù é ù ê ú ê ú ê ú ê ú ê ú ê ú Þ = = = = = = ê ú ê ú ê ú ê ú ê ú ê ú ê ú ê ú ê ú ê ú ê ú ê ú - ë û ë û ë û ë û ë û ë û e e e 1 1 2 2 1 2 3 3 1 3 0 1 3 0 3 3 1 0 3 1 0 3 0 0 2 0 0 2 2 x x x A A x x x x x + é ù é ù é ù é ù ê ú ê ú ê ú ê ú = Þ = = + ê ú ê ú ê ú ê ú ê ú ê ú ê ú ê ú - - - ë û ë û ë û ë û x n Thus, the above linear transformation T is with the following corresponding standard matrix A such that T ( x ) = A x ※ The statement on Slide 7.18 is valid because for any linear transformation T : V → V , there is a corresponding square matrix such that T ( x ) = A x Consequently, the eignvalues and eigenvectors of a linear transformation T are in essence the eigenvalues and eigenvectors of the corresponding square matrix A