Introductory Linear Algebra/Eigenvalues and eigenvectors


Motivation edit

Before discussing eigenvalues, eigenvectors and diagonalization, we provide some motivations to them.

Example. (Formula of power of a diagonal matrix) Let  . Then,   for each positive integer  , since

 
and we can prove the formula of power of a diagonal matrix by induction.

Example. Let   and  . Then, it can be computed that  . Let  . Then,

 

We can see from this example that for some special matrices, their powers can be computed conveniently, by expressing in the form of   in which   is invertible matrix and   is diagonal matrix.

Naturally, given a matrix, we would be interested in knowing whether it can be expressed in the form of  , and if it can, what are   and  , so that we can compute its power conveniently. This is the main objective of this chapter.

Eigenvalues, eigenvectors and diagonalization edit

In view of the motivation section, we have the following definition.

Definition. (Diagonalizable matrix) A square matrix   is diagonalizable if there exists an invertible matrix   such that   is a diagonal matrix.

Remark. An equivalent condition is   for some diagonal matrix   and invertible matrix  , which matches with the form in the motivation section. So, if a matrix is diagonalizable, we can compute its power conveniently.

Example. The matrix   is diagonalizable, since there exists   such that   is a diagonal matrix (which is  ). Also, there exists   such that  .

 

Exercise.

Select all diagonalizable matrix (matrices).

A zero matrix.
 
A diagonal matrix.


The following are important and general concepts, which is related to diagonalizability in some sense.

Definition. (Eigenvector and eigenvalue) Let   be a square matrix. A nonzero vector   is an eigenvector of   if there exists a scalar   such that  . In that case,   is an eigenvalue of   corresponding to eigenvector  .

Remark.

  •   means that multiplying the vector   by the matrix   is equivalent to multiplying it by a scalar (scaling of vector)
  • the prefix eigen- has the meanings of 'own', 'proper', and 'characteristic'

Example. (Eigenvectors of identity matrix) Each vector   is an eigenvector of  , since

 
for each vector  , and their corresponding eigenvalues are all  .
 

Exercise.

Select all correct statement(s).

If   is an eigenvector of invertible matrix  , then it is also an eigenvector of  .
If   is an eigenvalue of  , then   is an eigenvalue of  .
Each vector   is an eigenvctor of the zero matrix  .
Zero vector is an eigenvector of each square matrix.
If there exists an eigenvector for a matrix, then there are infinitely many eigenvectors of that matrix.


The following theorem relates diagonalizable matrix with eigenvectors and eigenvalues.

Theorem. (Diagonalization) Let   be an   matrix. Then,   is diagonalizable if and only if   has   linearly independent eigenvectors. If   are linearly independent eigenvectors of   corresponding to the eigenvalues   (some of them are possibly the same), we can define an invertible matrix   with columns   and a diagonal matrix   with diagonal entries   such that

 

Proof. In the following, we use   to denote the matrix with columns  , in this order.

 
We have now proved that   are eigenvectors. It remains to prove that they are linearly independent, which is true since they are linearly independent if and only if   is invertible by the proposition about relationship between invertibility and linear independence.  

Remark.

  • we can put the eigenvectors into   as columns in arbitrary orders, as long as we put the eigenvalues into the corresponding column of   e.g. we may put   into the 3rd column of  , but we need to put   into the 3rd column of  
  • it follows that the expression for diagonalization is not unique, and there are actually infinitely many expressions
  • we have
     
    by definition of matrix multiplication. E.g.,
     

Then, we will introduce a convenient way to find eigenvalues. Before this, we introduce a terminology which is related to this way of finding eigenvalues.

Definition. (Characteristic polynomial) Let   be an   matrix. The characteristic polynomial of   in the variable   is the polynomial  .

Remark.

  • we can use arbitrary letter for the variable.
  • equivalently, the characteristic polynomial of   is the determinant of   with its diagonal entries subtracted by  

Example. The characteristic polynomial of   is

 

Proposition. (Equivalent condition for eigenvalue) Let   be an   matrix. Then,   is an eigenvalue of   if and only if  , i.e. it is a root of the characteristic polynomial of  .

Proof.

 

 

Then, we will introduce a concept which is related to eigenvector.

 

Exercise.

Select all correct statement(s).

A square matrix has   different eigenvalues if its characteristic polynomial has   roots.
If   with size   has two linearly independent eigenvectors   corresponding to the eigenvalues  , then we can define an invertible matrix   and diagonal matrix   such that  .
If   with size   has two linearly independent eigenvectors, then we can define an invertible matrix   and a diagonal matrix   such that  .


Definition. (Eigenspace) Let   be an   matrix. Suppose   is an eigenvalue of  . Then, the null space  , denoted by  , is the eigenspace of   corresponding to  .

Remark.

  • since the null space is a subspace of  , the eigensapce is also a subspace of  
  •   consists of the zero vector (since it is subspace) and all eigenvectors corresponding to   since

 

After introducing these terminologies and concepts, we have the following algorithmic procedure for a diagonalization of an   matrix:

  1. compute all eigenvalues of   by solving  
  2. for each eigenvalue   of  , find a basis   for the eigenspace   respectively
  3. if   together contain   vectors   (if not,   is not diagonalizable), define
     
  4. we have   in which   is a diagonal matrix whose diagonal entries are eigenvalues corresponding to  

Remark.

  • it can be proved that eigenvectors of   corresponding to distinct eigenvalues are linearly independent (its proof is skipped here)
  • so the columns of   are linearly independent, thus   is invertible
  • if   has   different eigenvalues, then   is diagonalizable [1], since there are   bases corresponding to the   eigenvalues, which together contain   vectors  .
  • there are infinitely many possible bases for each eigenvalue, but we only need one of them

Example. (Diagonalization of   matrix) Recall the example in the motivation section, it is given that the matrix   is diagonalizable, and its expression in the form of   is also given. We will use the above procedure to derive the given expression.

First,

 
So, the eigenvalues of the matrix are   and  .

For the eigenvalue  , since  , and it can be proved that its general solution is  , so a basis for   is  

For the eigenvalue  , since  , and it can be proved that its general solution is  , so a basis for   is  

Then, we let   (since the two bases together contain two vectors), and   Then, we can compute that   Therefore, we have

 
which is the same as the given form in the example in motivation section. In general, if we have  ,
 
which is illustrated in the example in the motivation section. From the example in motivation section,
 

Example. (Diagonalization of   matrix) Consider the matrix   (It is not  ). We would like to find a formula for  . First,

 
So, the eigenvalues of the matrix are   and  .

For the eigenvalue  , since

 
(There are two independent unknowns, so the dimension of each basis for the eigenspace is  , i.e. there should be two vectors in each basis) a basis for   is  .

For the eigenvalue  , since

 
a basis for   is  .

Then, we let  , (since the two bases together contain three vectors)   (we have two eigenvectors corresponding to the eigenvalue  , so this eigenvalue is repeated two times). Then, we can compute that  . It follows that

 
and
 
This is an interesting result.

Example. (Complex eigenvalues) Let  [2]

 
Since the eigenvalues are all complex (and thus there does not exist any corresponding real eigenvector),   is not diagonalizable over real matrices. On the other hand,   is diagonalizable over complex matrices, but diagonalization over complex matrices is not our main focus in this book, and we have not defined the operation of complex matrices. So, an expression of   in the form of   is put in the following for reference:
 

Example. (Non-diagonalizable matrix) Consider the matrix   (It is a nilpotent matrix, satisfying  ).

First, since

 
the only eigenvalue is  .

For eigenvalue  , since

 
So, a basis for   is  . Since it only contains one vector, while the size of the matrix is  ,   is not diagonalizable.
 

Exercise.

1 Compute  .

 .
 .
 .
 .

2 Select all correct statement(s).

Over real matrices, eigenspace can be a zero space, i.e. only consists of zero vector.
Eigenspace must consist of infinitely many eigenvectors.
Each basis of the eigenspace consists of linearly independent vectors.
Suppose through diagonalization, matrix   can be expressed as   in which   is an invertible matrix and   is a diagonal matrix, then,   is a diagonal matrix for each positive integer  .


In the following, we will discuss some mathematical applications of diagonalization, namely deducing the sequence formula, and solving system of ordinary differential equations (ODEs).

Example. (Fibonacci sequence) Consider the Fibonacci sequence  , in which  ,   and   for each nonngeative integer  . For each nonnegative integer  , this recurrence relation can be described as

 

Let  . Then,

 

To obtain the expression for  , it suffices to find the formula for  , and we may find it by diagonalization.

Since

 
Let   be the golden ratio, and   be the conjugate of the golden ratio.

For eigenvalue  , since for

 
we can transform the augmented matrix representing this SLE to RREF as follows:
 
[3] So, the general solution is  , and thus a basis for   is  .

For eigenvalue  , since

 
and the RREF of the augmented matrix representing this SLE is
 
by symmetry[4]. So, the general solution is  , and thus a basis for   is  .

Then, we let  ,  . We can compute that   Then,  , and thus

 
Finally, we have
 
Thus,
 
in which   and  .
 

Exercise. Define a sequence   in which   and   for each nonnegative integer  .

1 Which of the following is (are) correct formula(s) of  ?

 .
 .
 .
 .

2 Define another sequence   in which   and   for each nonnegative integer  . Which of the following is (are) correct formula(s) of  ?

 .
 .
 .
 .
 .


Example. (System of ODEs) Consider the system of ODEs

 
with initial conditions   when  .

Using the dot notation for differentiation, the system can be rewritten as

 
in which  . Suppose we can write   in which   is an invertible matrix and   is a diagonal matrix. Let   in which   are some real numbers. Also, let  , which implies   and  , and   . It follows that   and  . Thus,
 
Let  , then the system can be simplified to
 
in which   are arbitrary constants, and  .

Then, we find   by diagonalizing  :

 
For eigenvalue  ,
 
and the general solution is  , so a basis for  .

For eigenvalue  ,

 
and the general solution is  , so a basis for  .

Then, we let   and  . It follows that  . Then,  .

It follows that   and  , and so   and  . Imposing the initial condition   when  ,

 
when  , which implies   and  . So,
 
Thus, the solution to this system of ODEs is
 
 

Exercise.

1 Solve the following system of ODEs with initial condition   when  :

 

 .
 .
 .
 .
 .

2 Find   such that the following system of ODEs is inconsistent.

 

No such   exists.
 
 
  can be arbitrary real numbers.

3 Solve the following system with initial condition   when  :

 

It is inconsistent.
 
 
 
 
 
 



  1. but even if   has strictly less than   eigenvalues,   can still be diagonalizable. Actually,   can at most have   different eigenvalues, since the characteristic polynomial in   is a degree   polynomial in  , which has   roots (some of them may be repeated), by fundamental theorem of algebra
  2. It is the matrix representation of the complex number  .
  3.   since  
  4. In particular,  , since both   satisfy the equation  .