Linear Algebra/Spectral Theorem

Given a Hermitian matrix ${\displaystyle A}$, ${\displaystyle A}$ is always diagonalizable. It is also the case that all eigenvalues of ${\displaystyle A}$ are real, and that all eigenvectors are mutually orthogonal. This is given by the "Spectral Theorem":

The Spectral Theorem

Given any ${\displaystyle n\times n}$ Hermitian matrix ${\displaystyle A}$, there exists an ${\displaystyle n\times n}$ unitary matrix ${\displaystyle U}$, and an ${\displaystyle n\times n}$ diagonal matrix of real values ${\displaystyle \Lambda }$ such that ${\displaystyle A=U\Lambda U^{H}}$

The columns of ${\displaystyle U={\begin{pmatrix}{\vec {u}}_{1}&{\vec {u}}_{2}&\dots &{\vec {u}}_{n}\end{pmatrix}}}$ are the eigenvectors of ${\displaystyle U}$, and the diagonal entries of ${\displaystyle \Lambda ={\begin{pmatrix}\lambda _{1}&0&\cdots &0\\0&\lambda _{2}&\cdots &0\\\vdots &\vdots &\ddots &\vdots \\0&0&\cdots &\lambda _{n}\end{pmatrix}}}$ are the corresponding eigenvalues.

In essence ${\displaystyle A}$ can be decomposed into a "spectrum" of rank 1 projections: ${\displaystyle A=\sum _{i=1}^{n}\lambda _{i}({\vec {u}}_{i}{\vec {u}}_{i}^{H})}$

The spectral theorem can in fact be proven without the need for the characteristic polynomial of ${\displaystyle A}$, or any of the derivative theorems.

Proof of the Spectral Theorem

The proof will proceed by using induction on ${\displaystyle n}$.

Base Case ${\displaystyle n=1}$

When ${\displaystyle A={\begin{pmatrix}a_{1,1}\end{pmatrix}}}$, it must be the case that ${\displaystyle a_{1,1}}$ is real, or else ${\displaystyle A}$ is not Hermitian. The spectral decomposition is then simply ${\displaystyle A={\begin{pmatrix}1\end{pmatrix}}{\begin{pmatrix}a_{1,1}\end{pmatrix}}{\begin{pmatrix}1\end{pmatrix}}^{H}}$

Inductive Case ${\displaystyle n\geq 2}$

Let ${\displaystyle {\vec {e}}_{i}}$ denote the ${\displaystyle i^{\text{th}}}$ standard basis vector of ${\displaystyle \mathbb {C} ^{n}}$. Let ${\displaystyle \mathbf {0} _{k\times m}}$ denote a ${\displaystyle k\times m}$ matrix of 0s.

Let ${\displaystyle {\vec {u}}_{1}}$ be a unit length vector that maximizes ${\displaystyle {\vec {u}}_{1}^{H}A{\vec {u}}_{1}}$ (recall that ${\displaystyle {\vec {u}}_{1}^{H}A{\vec {u}}_{1}}$ is always real), and let ${\displaystyle \lambda _{1}={\vec {u}}_{1}^{H}A{\vec {u}}_{1}}$. Let ${\displaystyle U_{1}}$ be a unitary matrix where the first column is ${\displaystyle {\vec {u}}_{1}}$: ${\displaystyle U_{1}{\vec {e}}_{1}={\vec {u}}_{1}}$.

${\displaystyle A=U_{1}A_{1}U_{1}^{H}}$ where ${\displaystyle A_{1}=U_{1}^{H}AU_{1}}$. It will now be shown that ${\displaystyle A_{1}}$ has the form ${\displaystyle A_{1}={\begin{pmatrix}\lambda _{1}&\mathbf {0} _{1\times (n-1)}\\\mathbf {0} _{(n-1)\times 1}&A_{\text{reduced}}\end{pmatrix}}}$

${\displaystyle {\vec {e}}_{1}^{H}A_{1}{\vec {e}}_{1}={\vec {u}}_{1}^{H}A{\vec {u}}_{1}=\lambda _{1}}$ so that the (1,1) entry of ${\displaystyle A_{1}}$ is ${\displaystyle \lambda _{1}}$. It will now be shown that the first row and column of ${\displaystyle A_{1}}$ is filled with 0s except for the first entry. For an arbitary ${\displaystyle i=2,3,\dots ,n}$, consider the parameterized unit vector ${\displaystyle {\vec {u}}_{t}(t)=(\cos t){\vec {e}}_{1}+(\sin t){\vec {e}}_{i}}$.

${\displaystyle {\vec {u}}_{t}(t)^{H}A_{1}{\vec {u}}_{t}(t)=((\cos t){\vec {e}}_{1}+(\sin t){\vec {e}}_{i})^{H}A_{1}((\cos t){\vec {e}}_{1}+(\sin t){\vec {e}}_{i})}$ ${\displaystyle =\lambda _{1}\cos ^{2}t+(a_{(i,1)}+a_{(1,i)})\cos t\sin t+a_{(i,i)}\sin ^{2}t}$ ${\displaystyle =\lambda _{1}\cos ^{2}t+2\Re (a_{(i,1)})\cos t\sin t+a_{(i,i)}\sin ^{2}t}$ where ${\displaystyle a_{(i,j)}}$ denotes the ${\displaystyle (i,j)}$ entry of ${\displaystyle A_{1}}$ (${\displaystyle \Re }$ and ${\displaystyle \Im }$ denote the real and imaginary components respectively).

${\displaystyle {\frac {d}{dt}}({\vec {u}}_{t}(t)^{H}A_{1}{\vec {u}}_{t}(t)){\bigg |}_{t=0}=2\Re (a_{(i,1)})}$. Unit vector ${\displaystyle {\vec {u}}={\vec {u}}_{1}}$ maximizes ${\displaystyle {\vec {u}}^{H}A{\vec {u}}}$ which implies that ${\displaystyle {\vec {u}}={\vec {u}}_{t}(0)={\vec {e}}_{1}}$ is the unit vector that maximizes ${\displaystyle {\vec {u}}^{H}A_{1}{\vec {u}}}$. Therefore ${\displaystyle {\frac {d}{dt}}({\vec {u}}_{t}(t)^{H}A_{1}{\vec {u}}_{t}(t)){\bigg |}_{t=0}=0}$ which gives ${\displaystyle \Re (a_{(i,1)})=0}$.

Now consider the parameterized unit vector ${\displaystyle {\vec {u}}_{t}(t)=(\cos t){\vec {e}}_{1}+i(\sin t){\vec {e}}_{i}}$.

${\displaystyle {\vec {u}}_{t}(t)^{H}A_{1}{\vec {u}}_{t}(t)=((\cos t){\vec {e}}_{1}+i(\sin t){\vec {e}}_{i})^{H}A_{1}((\cos t){\vec {e}}_{1}+i(\sin t){\vec {e}}_{i})}$ ${\displaystyle =\lambda _{1}\cos ^{2}t+(-ia_{(i,1)}+ia_{(1,i)})\cos t\sin t+a_{(i,i)}\sin ^{2}t}$ ${\displaystyle =\lambda _{1}\cos ^{2}t+2\Im (a_{(i,1)})\cos t\sin t+a_{(i,i)}\sin ^{2}t}$

${\displaystyle {\frac {d}{dt}}({\vec {u}}_{t}(t)^{H}A_{1}{\vec {u}}_{t}(t)){\bigg |}_{t=0}=2\Im (a_{(i,1)})}$ so ${\displaystyle {\frac {d}{dt}}({\vec {u}}_{t}(t)^{H}A_{1}{\vec {u}}_{t}(t)){\bigg |}_{t=0}=0\implies \Im (a_{(i,1)})=0}$. Therefore ${\displaystyle a_{(i,1)}=a_{(1,i)}=0}$. It has been proven that the first row and column of ${\displaystyle A_{1}}$ is filled with 0s except for the first entry.

${\displaystyle A_{1}}$ has the form ${\displaystyle A_{1}={\begin{pmatrix}\lambda _{1}&\mathbf {0} _{1\times (n-1)}\\\mathbf {0} _{(n-1)\times 1}&A_{\text{reduced}}\end{pmatrix}}}$, and by an inductive argument, ${\displaystyle A_{\text{reduced}}}$ has the spectral decomposition ${\displaystyle U_{r}\Lambda _{r}U_{r}^{H}}$.

Therefore ${\displaystyle A=U\Lambda U^{H}}$ where ${\displaystyle U=U_{1}{\begin{pmatrix}1&\mathbf {0} _{1\times (n-1)}\\\mathbf {0} _{(n-1)\times 1}&U_{r}\end{pmatrix}}}$ and ${\displaystyle \Lambda ={\begin{pmatrix}\lambda _{1}&\mathbf {0} _{1\times (n-1)}\\\mathbf {0} _{(n-1)\times 1}&\Lambda _{r}\end{pmatrix}}}$