Linear Algebra/Projection Onto a Subspace/Solutions

Solutions

This exercise is recommended for all readers.

Problem 1

Project the vectors onto $M$ along $N$.

1. ${\begin{pmatrix}3\\-2\end{pmatrix}},\quad M=\{{\begin{pmatrix}x\\y\end{pmatrix}}\,{\big |}\,x+y=0\},\quad N=\{{\begin{pmatrix}x\\y\end{pmatrix}}\,{\big |}\,-x-2y=0\}$
2. ${\begin{pmatrix}1\\2\end{pmatrix}},\quad M=\{{\begin{pmatrix}x\\y\end{pmatrix}}\,{\big |}\,x-y=0\},\quad N=\{{\begin{pmatrix}x\\y\end{pmatrix}}\,{\big |}\,2x+y=0\}$
3. ${\begin{pmatrix}3\\0\\1\end{pmatrix}},\quad M=\{{\begin{pmatrix}x\\y\\z\end{pmatrix}}\,{\big |}\,x+y=0\},\quad N=\{c\cdot {\begin{pmatrix}1\\0\\1\end{pmatrix}}\,{\big |}\,c\in \mathbb {R} \}$

Answer

1. When bases for the subspaces

   $B_{M}=\langle {\begin{pmatrix}1\\-1\end{pmatrix}}\rangle \qquad B_{N}=\langle {\begin{pmatrix}2\\-1\end{pmatrix}}\rangle$

   are concatenated

   $B=B_{M}\!{\mathbin {{}^{\frown }}}\!B_{N}=\langle {\begin{pmatrix}1\\-1\end{pmatrix}},{\begin{pmatrix}2\\-1\end{pmatrix}}\rangle$

   and the given vector is represented

   ${\begin{pmatrix}3\\-2\end{pmatrix}}=1\cdot {\begin{pmatrix}1\\-1\end{pmatrix}}+1\cdot {\begin{pmatrix}2\\-1\end{pmatrix}}$

   then the answer comes from retaining the $M$ part and dropping the $N$ part.

   ${\mbox{proj}}_{M,N}({\begin{pmatrix}3\\-2\end{pmatrix}})={\begin{pmatrix}1\\-1\end{pmatrix}}$

2. When the bases

   $B_{M}=\langle {\begin{pmatrix}1\\1\end{pmatrix}}\rangle \qquad B_{N}\langle {\begin{pmatrix}1\\-2\end{pmatrix}}\rangle$

   are concatenated, and the vector is represented,

   ${\begin{pmatrix}1\\2\end{pmatrix}}=(4/3)\cdot {\begin{pmatrix}1\\1\end{pmatrix}}-(1/3)\cdot {\begin{pmatrix}1\\-2\end{pmatrix}}$

   then retaining only the $M$ part gives this answer.

   ${\mbox{proj}}_{M,N}({\begin{pmatrix}1\\2\end{pmatrix}})={\begin{pmatrix}4/3\\4/3\end{pmatrix}}$

3. With these bases

   $B_{M}=\langle {\begin{pmatrix}1\\-1\\0\end{pmatrix}},{\begin{pmatrix}0\\0\\1\end{pmatrix}}\rangle \qquad B_{N}=\langle {\begin{pmatrix}1\\0\\1\end{pmatrix}}\rangle$

   the representation with respect to the concatenation is this.

   ${\begin{pmatrix}3\\0\\1\end{pmatrix}}=0\cdot {\begin{pmatrix}1\\-1\\0\end{pmatrix}}-2\cdot {\begin{pmatrix}0\\0\\1\end{pmatrix}}+3\cdot {\begin{pmatrix}1\\0\\1\end{pmatrix}}$

   and so the projection is this.

   ${\mbox{proj}}_{M,N}({\begin{pmatrix}3\\0\\1\end{pmatrix}})={\begin{pmatrix}0\\0\\-2\end{pmatrix}}$

This exercise is recommended for all readers.

Problem 2

Find $M^{\perp }$.

1. $M=\{{\begin{pmatrix}x\\y\end{pmatrix}}\,{\big |}\,x+y=0\}$
2. $M=\{{\begin{pmatrix}x\\y\end{pmatrix}}\,{\big |}\,-2x+3y=0\}$
3. $M=\{{\begin{pmatrix}x\\y\end{pmatrix}}\,{\big |}\,x-y=0\}$
4. $M=\{{\vec {0}}\,\}$
5. $M=\{{\begin{pmatrix}x\\y\end{pmatrix}}\,{\big |}\,x=0\}$
6. $M=\{{\begin{pmatrix}x\\y\\z\end{pmatrix}}\,{\big |}\,-x+3y+z=0\}$
7. $M=\{{\begin{pmatrix}x\\y\\z\end{pmatrix}}\,{\big |}\,x=0{\text{ and }}y+z=0\}$

Answer

As in Example 3.5, we can simplify the calculation by just finding the space of vectors perpendicular to all the vectors in $M$'s basis.

1. Parametrizing to get

   $M=\{c\cdot {\begin{pmatrix}-1\\1\end{pmatrix}}\,{\big |}\,c\in \mathbb {R} \}$

   gives that

   $M^{\perp }\{{\begin{pmatrix}u\\v\end{pmatrix}}\,{\big |}\,0={\begin{pmatrix}u\\v\end{pmatrix}}\cdot {\begin{pmatrix}-1\\1\end{pmatrix}}\}=\{{\begin{pmatrix}u\\v\end{pmatrix}}\,{\big |}\,0=-u+v\}$

   Parametrizing the one-equation linear system gives this description.

   $M^{\perp }=\{k\cdot {\begin{pmatrix}1\\1\end{pmatrix}}\,{\big |}\,k\in \mathbb {R} \}$

2. As in the answer to the prior part, $M$ can be described as a span

   $M=\{c\cdot {\begin{pmatrix}3/2\\1\end{pmatrix}}\,{\big |}\,c\in \mathbb {R} \}\qquad B_{M}=\langle {\begin{pmatrix}3/2\\1\end{pmatrix}}\rangle$

   and then $M^{\perp }$ is the set of vectors perpendicular to the one vector in this basis.

   $M^{\perp }=\{{\begin{pmatrix}u\\v\end{pmatrix}}\,{\big |}\,(3/2)\cdot u+1\cdot v=0\}=\{k\cdot {\begin{pmatrix}-2/3\\1\end{pmatrix}}\,{\big |}\,k\in \mathbb {R} \}$

3. Parametrizing the linear requirement in the description of $M$ gives this basis.

   $M=\{c\cdot {\begin{pmatrix}1\\1\end{pmatrix}}\,{\big |}\,c\in \mathbb {R} \}\qquad B_{M}=\langle {\begin{pmatrix}1\\1\end{pmatrix}}\rangle$

   Now, $M^{\perp }$ is the set of vectors perpendicular to (the one vector in) $B_{M}$.

   $M^{\perp }=\{{\begin{pmatrix}u\\v\end{pmatrix}}\,{\big |}\,u+v=0\}=\{k\cdot {\begin{pmatrix}-1\\1\end{pmatrix}}\,{\big |}\,k\in \mathbb {R} \}$

   (By the way, this answer checks with the first item in this question.)
4. Every vector in the space is perpendicular to the zero vector so $M^{\perp }=\mathbb {R} ^{n}$.
5. The appropriate description and basis for $M$ are routine.

   $M=\{y\cdot {\begin{pmatrix}0\\1\end{pmatrix}}\,{\big |}\,y\in \mathbb {R} \}\qquad B_{M}=\langle {\begin{pmatrix}0\\1\end{pmatrix}}\rangle$

   Then

   $M^{\perp }=\{{\begin{pmatrix}u\\v\end{pmatrix}}\,{\big |}\,0\cdot u+1\cdot v=0\}=\{k\cdot {\begin{pmatrix}1\\0\end{pmatrix}}\,{\big |}\,k\in \mathbb {R} \}$

   and so $(y{\text{-axis}})^{\perp }=x{\text{-axis}}$.
6. The description of $M$ is easy to find by parametrizing.

   $M=\{c\cdot {\begin{pmatrix}3\\1\\0\end{pmatrix}}+d\cdot {\begin{pmatrix}1\\0\\1\end{pmatrix}}\,{\big |}\,c,d\in \mathbb {R} \}\qquad B_{M}=\langle {\begin{pmatrix}3\\1\\0\end{pmatrix}},{\begin{pmatrix}1\\0\\1\end{pmatrix}}\rangle$

   Finding $M^{\perp }$ here just requires solving a linear system with two equations

   ${\begin{array}{*{3}{rc}r}3u&+&v&&&=&0\\u&&&+&w&=&0\end{array}}\;{\xrightarrow[{}]{-(1/3)\rho _{1}+\rho _{2}}}\;{\begin{array}{*{3}{rc}r}3u&+&v&&&=&0\\&&-(1/3)v&+&w&=&0\end{array}}$

   and parametrizing.

   $M^{\perp }=\{k\cdot {\begin{pmatrix}-1\\3\\1\end{pmatrix}}\,{\big |}\,k\in \mathbb {R} \}$

7. Here, $M$ is one-dimensional

   $M=\{c\cdot {\begin{pmatrix}0\\-1\\1\end{pmatrix}}\,{\big |}\,c\in \mathbb {R} \}\qquad B_{M}=\langle {\begin{pmatrix}0\\-1\\1\end{pmatrix}}\rangle$

   and as a result, $M^{\perp }$ is two-dimensional.

   $M^{\perp }=\{{\begin{pmatrix}u\\v\\w\end{pmatrix}}\,{\big |}\,0\cdot u-1\cdot v+1\cdot w=0\}=\{j\cdot {\begin{pmatrix}1\\0\\0\end{pmatrix}}+k\cdot {\begin{pmatrix}0\\1\\1\end{pmatrix}}\,{\big |}\,j,k\in \mathbb {R} \}$

Problem 3

This subsection shows how to project orthogonally in two ways, the method of Example 3.2 and 3.3, and the method of Theorem 3.8. To compare them, consider the plane $P$ specified by $3x+2y-z=0$ in $\mathbb {R} ^{3}$.

1. Find a basis for $P$.
2. Find $P^{\perp }$ and a basis for $P^{\perp }$.
3. Represent this vector with respect to the concatenation of the two bases from the prior item.

   ${\vec {v}}={\begin{pmatrix}1\\1\\2\end{pmatrix}}$

4. Find the orthogonal projection of ${\vec {v}}$ onto $P$ by keeping only the $P$ part from the prior item.
5. Check that against the result from applying Theorem 3.8.

Answer

1. Parametrizing the equation leads to this basis for $P$.

   $B_{P}=\langle {\begin{pmatrix}1\\0\\3\end{pmatrix}},{\begin{pmatrix}0\\1\\2\end{pmatrix}}\rangle$

2. Because $\mathbb {R} ^{3}$ is three-dimensional and $P$ is two-dimensional, the complement $P^{\perp }$ must be a line. Anyway, the calculation as in Example 3.5

   $P^{\perp }=\{{\begin{pmatrix}x\\y\\z\end{pmatrix}}\,{\big |}\,{\begin{pmatrix}1&0&3\\0&1&2\end{pmatrix}}{\begin{pmatrix}x\\y\\z\end{pmatrix}}={\begin{pmatrix}0\\0\end{pmatrix}}\;\}$

   gives this basis for $P^{\perp }$.

   $B_{P^{\perp }}=\langle {\begin{pmatrix}3\\2\\-1\end{pmatrix}}\rangle$

3. $\displaystyle {\begin{pmatrix}1\\1\\2\end{pmatrix}}=(5/14)\cdot {\begin{pmatrix}1\\0\\3\end{pmatrix}}+(8/14)\cdot {\begin{pmatrix}0\\1\\2\end{pmatrix}}+(3/14)\cdot {\begin{pmatrix}3\\2\\-1\end{pmatrix}}$
4. ${\mbox{proj}}_{P}({\begin{pmatrix}1\\1\\2\end{pmatrix}})={\begin{pmatrix}5/14\\8/14\\31/14\end{pmatrix}}$
5. The matrix of the projection

   ${\begin{array}{rl}{\begin{pmatrix}1&0\\0&1\\3&2\end{pmatrix}}{\bigl (}{\begin{pmatrix}1&0&3\\0&1&2\end{pmatrix}}{\begin{pmatrix}1&0\\0&1\\3&2\end{pmatrix}}{\bigr )}^{-1}{\begin{pmatrix}1&0&3\\0&1&2\end{pmatrix}}&={\begin{pmatrix}1&0\\0&1\\3&2\end{pmatrix}}{\begin{pmatrix}10&6\\6&5\end{pmatrix}}^{-1}{\begin{pmatrix}1&0&3\\0&1&2\end{pmatrix}}\\&={\frac {1}{14}}{\begin{pmatrix}5&-6&3\\-6&10&2\\3&2&13\end{pmatrix}}\end{array}}$

   when applied to the vector, yields the expected result.

   ${\frac {1}{14}}{\begin{pmatrix}5&-6&3\\-6&10&2\\3&2&13\end{pmatrix}}{\begin{pmatrix}1\\1\\2\end{pmatrix}}={\begin{pmatrix}5/14\\8/14\\31/14\end{pmatrix}}$

This exercise is recommended for all readers.

Problem 4

We have three ways to find the orthogonal projection of a vector onto a line, the Definition 1.1 way from the first subsection of this section, the Example 3.2 and 3.3 way of representing the vector with respect to a basis for the space and then keeping the $M$ part, and the way of Theorem 3.8. For these cases, do all three ways.

1. ${\vec {v}}={\begin{pmatrix}1\\-3\end{pmatrix}},\quad M=\{{\begin{pmatrix}x\\y\end{pmatrix}}\,{\big |}\,x+y=0\}$
2. ${\vec {v}}={\begin{pmatrix}0\\1\\2\end{pmatrix}},\quad M=\{{\begin{pmatrix}x\\y\\z\end{pmatrix}}\,{\big |}\,x+z=0{\text{ and }}y=0\}$

Answer

1. Parametrizing gives this.

   $M=\{c\cdot {\begin{pmatrix}-1\\1\end{pmatrix}}\,{\big |}\,c\in \mathbb {R} \}$

   For the first way, we take the vector spanning the line $M$ to be

   ${\vec {s}}={\begin{pmatrix}-1\\1\end{pmatrix}}$

   and the Definition 1.1 formula gives this.

   ${\mbox{proj}}_{[{\vec {s}}\,]}({\begin{pmatrix}1\\-3\end{pmatrix}})={\frac {{\begin{pmatrix}1\\-3\end{pmatrix}}\cdot {\begin{pmatrix}-1\\1\end{pmatrix}}}{{\begin{pmatrix}-1\\1\end{pmatrix}}\cdot {\begin{pmatrix}-1\\1\end{pmatrix}}}}\cdot {\begin{pmatrix}-1\\1\end{pmatrix}}={\frac {-4}{2}}\cdot {\begin{pmatrix}-1\\1\end{pmatrix}}={\begin{pmatrix}2\\-2\end{pmatrix}}$

   For the second way, we fix

   $B_{M}=\langle {\begin{pmatrix}-1\\1\end{pmatrix}}\rangle$

   and so (as in Example 3.5 and 3.6, we can just find the vectors perpendicular to all of the members of the basis)

   $M^{\perp }=\{{\begin{pmatrix}u\\v\end{pmatrix}}\,{\big |}\,-1\cdot u+1\cdot v=0\}=\{k\cdot {\begin{pmatrix}1\\1\end{pmatrix}}\,{\big |}\,k\in \mathbb {R} \}\qquad B_{M^{\perp }}=\langle {\begin{pmatrix}1\\1\end{pmatrix}}\rangle$

   and representing the vector with respect to the concatenation gives this.

   ${\begin{pmatrix}1\\-3\end{pmatrix}}=-2\cdot {\begin{pmatrix}-1\\1\end{pmatrix}}-1\cdot {\begin{pmatrix}1\\1\end{pmatrix}}$

   Keeping the $M$ part yields the answer.

   ${\mbox{proj}}_{M,M^{\perp }}({\begin{pmatrix}1\\-3\end{pmatrix}})={\begin{pmatrix}2\\-2\end{pmatrix}}$

   The third part is also a simple calculation (there is a $1\!\times \!1$ matrix in the middle, and the inverse of it is also $1\!\times \!1$)

   $A\left({{A}^{\rm {trans}}}A\right)^{-1}{{A}^{\rm {trans}}}={\begin{pmatrix}-1\\1\end{pmatrix}}\left({\begin{pmatrix}-1&1\end{pmatrix}}{\begin{pmatrix}-1\\1\end{pmatrix}}\right)^{-1}{\begin{pmatrix}-1&1\end{pmatrix}}={\begin{pmatrix}-1\\1\end{pmatrix}}{\begin{pmatrix}2\end{pmatrix}}^{-1}{\begin{pmatrix}-1&1\end{pmatrix}}$

   $={\begin{pmatrix}-1\\1\end{pmatrix}}{\begin{pmatrix}1/2\end{pmatrix}}{\begin{pmatrix}-1&1\end{pmatrix}}={\begin{pmatrix}-1\\1\end{pmatrix}}{\begin{pmatrix}-1/2&1/2\end{pmatrix}}={\begin{pmatrix}1/2&-1/2\\-1/2&1/2\end{pmatrix}}$

   which of course gives the same answer.

   ${\mbox{proj}}_{M}({\begin{pmatrix}1\\-3\end{pmatrix}})={\begin{pmatrix}1/2&-1/2\\-1/2&1/2\end{pmatrix}}{\begin{pmatrix}1\\-3\end{pmatrix}}={\begin{pmatrix}2\\-2\end{pmatrix}}$

2. Parametrization gives this.

   $M=\{c\cdot {\begin{pmatrix}-1\\0\\1\end{pmatrix}}\,{\big |}\,c\in \mathbb {R} \}$

   With that, the formula for the first way gives this.

   ${\frac {{\begin{pmatrix}0\\1\\2\end{pmatrix}}\cdot {\begin{pmatrix}-1\\0\\1\end{pmatrix}}}{{\begin{pmatrix}-1\\0\\1\end{pmatrix}}\cdot {\begin{pmatrix}-1\\0\\1\end{pmatrix}}}}\cdot {\begin{pmatrix}-1\\0\\1\end{pmatrix}}={\frac {2}{2}}\cdot {\begin{pmatrix}-1\\0\\1\end{pmatrix}}={\begin{pmatrix}-1\\0\\1\end{pmatrix}}$

   To proceed by the second method we find $M^{\perp }$,

   $M^{\perp }=\{{\begin{pmatrix}u\\v\\w\end{pmatrix}}\,{\big |}\,-u+w=0\}=\{j\cdot {\begin{pmatrix}1\\0\\1\end{pmatrix}}+k\cdot {\begin{pmatrix}0\\1\\0\end{pmatrix}}\,{\big |}\,j,k\in \mathbb {R} \}$

   find the representation of the given vector with respect to the concatenation of the bases $B_{M}$ and $B_{M^{\perp }}$

   ${\begin{pmatrix}0\\1\\2\end{pmatrix}}=1\cdot {\begin{pmatrix}-1\\0\\1\end{pmatrix}}+1\cdot {\begin{pmatrix}1\\0\\1\end{pmatrix}}+1\cdot {\begin{pmatrix}0\\1\\0\end{pmatrix}}$

   and retain only the $M$ part.

   ${\mbox{proj}}_{M}({\begin{pmatrix}0\\1\\2\end{pmatrix}})=1\cdot {\begin{pmatrix}-1\\0\\1\end{pmatrix}}={\begin{pmatrix}-1\\0\\1\end{pmatrix}}$

   Finally, for the third method, the matrix calculation

   $A\left({{A}^{\rm {trans}}}A\right)^{-1}{{A}^{\rm {trans}}}={\begin{pmatrix}-1\\0\\1\end{pmatrix}}{\bigl (}{\begin{pmatrix}-1&0&1\end{pmatrix}}{\begin{pmatrix}-1\\0\\1\end{pmatrix}}{\bigr )}^{-1}{\begin{pmatrix}-1&0&1\end{pmatrix}}={\begin{pmatrix}-1\\0\\1\end{pmatrix}}{\begin{pmatrix}2\end{pmatrix}}^{-1}{\begin{pmatrix}-1&0&1\end{pmatrix}}$

   $={\begin{pmatrix}-1\\0\\1\end{pmatrix}}{\begin{pmatrix}1/2\end{pmatrix}}{\begin{pmatrix}-1&0&1\end{pmatrix}}={\begin{pmatrix}-1\\0\\1\end{pmatrix}}{\begin{pmatrix}-1/2&0&1/2\end{pmatrix}}={\begin{pmatrix}1/2&0&-1/2\\0&0&0\\-1/2&0&1/2\end{pmatrix}}$

   followed by matrix-vector multiplication

   ${\mbox{proj}}_{M}({\begin{pmatrix}0\\1\\2\end{pmatrix}}){\begin{pmatrix}1/2&0&-1/2\\0&0&0\\-1/2&0&1/2\end{pmatrix}}{\begin{pmatrix}0\\1\\2\end{pmatrix}}={\begin{pmatrix}-1\\0\\1\end{pmatrix}}$

   gives the answer.

Problem 5

Check that the operation of Definition 3.1 is well-defined. That is, in Example 3.2 and 3.3, doesn't the answer depend on the choice of bases?

Answer

No, a decomposition of vectors ${\vec {v}}={\vec {m}}+{\vec {n}}$ into ${\vec {m}}\in M$ and ${\vec {n}}\in N$ does not depend on the bases chosen for the subspaces— this was shown in the Direct Sum subsection.

Problem 6

What is the orthogonal projection onto the trivial subspace?

Answer

The orthogonal projection of a vector onto a subspace is a member of that subspace. Since a trivial subspace has only one member, ${\vec {0}}$, the projection of any vector must equal ${\vec {0}}$.

Problem 7

What is the projection of ${\vec {v}}$ onto $M$ along $N$ if ${\vec {v}}\in M$?

Answer

The projection onto $M$ along $N$ of a ${\vec {v}}\in M$ is ${\vec {v}}$. Decomposing ${\vec {v}}={\vec {m}}+{\vec {n}}$ gives ${\vec {m}}={\vec {v}}$ and ${\vec {n}}={\vec {0}}$, and dropping the $N$ part but retaining the $M$ part results in a projection of ${\vec {m}}={\vec {v}}$.

Problem 8

Show that if $M\subseteq \mathbb {R} ^{n}$ is a subspace with orthonormal basis $\langle {\vec {\kappa }}_{1},\ldots ,{\vec {\kappa }}_{n}\rangle$ then the orthogonal projection of ${\vec {v}}$ onto $M$ is this.

$({\vec {v}}\cdot {\vec {\kappa }}_{1})\cdot {\vec {\kappa }}_{1}+\dots +({\vec {v}}\cdot {\vec {\kappa }}_{n})\cdot {\vec {\kappa }}_{n}$

Answer

The proof of Lemma 3.7 shows that each vector ${\vec {v}}\in \mathbb {R} ^{n}$ is the sum of its orthogonal projections onto the lines spanned by the basis vectors.

${\vec {v}}={\mbox{proj}}_{[{\vec {\kappa }}_{1}]}({{\vec {v}}\,})+\dots +{\mbox{proj}}_{[{\vec {\kappa }}_{n}]}({{\vec {v}}\,})={\frac {{\vec {v}}\cdot {\vec {\kappa }}_{1}}{{\vec {\kappa }}_{1}\cdot {\vec {\kappa }}_{1}}}\cdot {\vec {\kappa }}_{1}+\dots +{\frac {{\vec {v}}\cdot {\vec {\kappa }}_{n}}{{\vec {\kappa }}_{n}\cdot {\vec {\kappa }}_{n}}}\cdot {\vec {\kappa }}_{n}$

Since the basis is orthonormal, the bottom of each fraction has ${\vec {\kappa }}_{i}\cdot {\vec {\kappa }}_{i}=1$.

This exercise is recommended for all readers.

Problem 9

Prove that the map $p:V\to V$ is the projection onto $M$ along $N$ if and only if the map ${\mbox{id}}-p$ is the projection onto $N$ along $M$. (Recall the definition of the difference of two maps: $({\mbox{id}}-p)\,({\vec {v}})={\mbox{id}}({\vec {v}})-p({\vec {v}})={\vec {v}}-p({\vec {v}})$.)

Answer

If $V=M\oplus N$ then every vector can be decomposed uniquely as ${\vec {v}}={\vec {m}}+{\vec {n}}$. For all ${\vec {v}}$ the map $p$ gives $p({\vec {v}})={\vec {m}}$ if and only if ${\vec {v}}-p({\vec {v}})={\vec {n}}$, as required.

This exercise is recommended for all readers.

Problem 10

Show that if a vector is perpendicular to every vector in a set then it is perpendicular to every vector in the span of that set.

Answer

Let ${\vec {v}}$ be perpendicular to every ${\vec {w}}\in S$. Then ${\vec {v}}\cdot (c_{1}{\vec {w}}_{1}+\dots +c_{n}{\vec {w}}_{n})={\vec {v}}\cdot (c_{1}{\vec {w}}_{1})+\dots +{\vec {v}}\cdot (c_{n}\cdot {\vec {w}}_{n})=c_{1}({\vec {v}}\cdot {\vec {w}}_{1})+\dots +c_{n}({\vec {v}}\cdot {\vec {w}}_{n})=c_{1}\cdot 0+\dots +c_{n}\cdot 0=0$.

Problem 11

True or false: the intersection of a subspace and its orthogonal complement is trivial.

Answer

True; the only vector orthogonal to itself is the zero vector.

Problem 12

Show that the dimensions of orthogonal complements add to the dimension of the entire space.

Answer

This is immediate from the statement in Lemma 3.7 that the space is the direct sum of the two.

This exercise is recommended for all readers.

Problem 13

Suppose that ${\vec {v}}_{1},{\vec {v}}_{2}\in \mathbb {R} ^{n}$ are such that for all complements $M,N\subseteq \mathbb {R} ^{n}$, the projections of ${\vec {v}}_{1}$ and ${\vec {v}}_{2}$ onto $M$ along $N$ are equal. Must ${\vec {v}}_{1}$ equal ${\vec {v}}_{2}$? (If so, what if we relax the condition to: all orthogonal projections of the two are equal?)

Answer

The two must be equal, even only under the seemingly weaker condition that they yield the same result on all orthogonal projections. Consider the subspace $M$ spanned by the set $\{{\vec {v}}_{1},{\vec {v}}_{2}\}$. Since each is in $M$, the orthogonal projection of ${\vec {v}}_{1}$ onto $M$ is ${\vec {v}}_{1}$ and the orthogonal projection of ${\vec {v}}_{2}$ onto $M$ is ${\vec {v}}_{2}$. For their projections onto $M$ to be equal, they must be equal.

This exercise is recommended for all readers.

Problem 14

Let $M,N$ be subspaces of $\mathbb {R} ^{n}$. The perp operator acts on subspaces; we can ask how it interacts with other such operations.

1. Show that two perps cancel: $(M^{\perp })^{\perp }=M$.
2. Prove that $M\subseteq N$ implies that $N^{\perp }\subseteq M^{\perp }$.
3. Show that $(M+N)^{\perp }=M^{\perp }\cap N^{\perp }$.

Answer

1. We will show that the sets are mutually inclusive, $M\subseteq (M^{\perp })^{\perp }$ and $(M^{\perp })^{\perp }\subseteq M$. For the first, if ${\vec {m}}\in M$ then by the definition of the perp operation, ${\vec {m}}$ is perpendicular to every ${\vec {v}}\in M^{\perp }$, and therefore (again by the definition of the perp operation) ${\vec {m}}\in (M^{\perp })^{\perp }$. For the other direction, consider ${\vec {v}}\in (M^{\perp })^{\perp }$. Lemma 3.7's proof shows that $\mathbb {R} ^{n}=M\oplus M^{\perp }$ and that we can give an orthogonal basis for the space $\langle {\vec {\kappa }}_{1},\dots ,{\vec {\kappa }}_{k},{\vec {\kappa }}_{k+1},\dots ,{\vec {\kappa }}_{n}\rangle$ such that the first half $\langle {\vec {\kappa }}_{1},\dots ,{\vec {\kappa }}_{k}\rangle$ is a basis for $M$ and the second half is a basis for $M^{\perp }$. The proof also checks that each vector in the space is the sum of its orthogonal projections onto the lines spanned by these basis vectors.

   ${\vec {v}}={\mbox{proj}}_{[{\vec {\kappa }}_{1}]}({{\vec {v}}\,})+\dots +{\mbox{proj}}_{[{\vec {\kappa }}_{n}]}({{\vec {v}}\,})$

   Because ${\vec {v}}\in (M^{\perp })^{\perp }$, it is perpendicular to every vector in $M^{\perp }$, and so the projections in the second half are all zero. Thus ${\vec {v}}={\mbox{proj}}_{[{\vec {\kappa }}_{1}]}({{\vec {v}}\,})+\dots +{\mbox{proj}}_{[{\vec {\kappa }}_{k}]}({{\vec {v}}\,})$, which is a linear combination of vectors from $M$, and so ${\vec {v}}\in M$. (Remark. Here is a slicker way to do the second half: write the space both as $M\oplus M^{\perp }$ and as $M^{\perp }\oplus (M^{\perp })^{\perp }$. Because the first half showed that $M\subseteq (M^{\perp })^{\perp }$ and the prior sentence shows that the dimension of the two subspaces $M$ and $(M^{\perp })^{\perp }$ are equal, we can conclude that $M$ equals $(M^{\perp })^{\perp }$.)
2. Because $M\subseteq N$, any ${\vec {v}}$ that is perpendicular to every vector in $N$ is also perpendicular to every vector in $M$. But that sentence simply says that $N^{\perp }\subseteq M^{\perp }$.
3. We will again show that the sets are equal by mutual inclusion. The first direction is easy; any ${\vec {v}}$ perpendicular to every vector in $M+N=\{{\vec {m}}+{\vec {n}}\,{\big |}\,{\vec {m}}\in M,\,{\vec {n}}\in N\}$ is perpendicular to every vector of the form ${\vec {m}}+{\vec {0}}$ (that is, every vector in $M$) and every vector of the form ${\vec {0}}+{\vec {n}}$ (every vector in $N$), and so $(M+N)^{\perp }\subseteq M^{\perp }\cap N^{\perp }$. The second direction is also routine; any vector ${\vec {v}}\in M^{\perp }\cap N^{\perp }$ is perpendicular to any vector of the form $c{\vec {m}}+d{\vec {n}}$ because ${\vec {v}}\cdot (c{\vec {m}}+d{\vec {n}})=c\cdot ({\vec {v}}\cdot {\vec {m}})+d\cdot ({\vec {v}}\cdot {\vec {n}})=c\cdot 0+d\cdot 0=0$.

This exercise is recommended for all readers.

Problem 15

The material in this subsection allows us to express a geometric relationship that we have not yet seen between the rangespace and the nullspace of a linear map.

1. Represent $f:\mathbb {R} ^{3}\to \mathbb {R}$ given by

   ${\begin{pmatrix}v_{1}\\v_{2}\\v_{3}\end{pmatrix}}\mapsto 1v_{1}+2v_{2}+3v_{3}$

   with respect to the standard bases and show that

   ${\begin{pmatrix}1\\2\\3\end{pmatrix}}$

   is a member of the perp of the nullspace. Prove that ${\mathcal {N}}(f)^{\perp }$ is equal to the span of this vector.
2. Generalize that to apply to any $f:\mathbb {R} ^{n}\to \mathbb {R}$.
3. Represent $f:\mathbb {R} ^{3}\to \mathbb {R} ^{2}$

   ${\begin{pmatrix}v_{1}\\v_{2}\\v_{3}\end{pmatrix}}\mapsto {\begin{pmatrix}1v_{1}+2v_{2}+3v_{3}\\4v_{1}+5v_{2}+6v_{3}\end{pmatrix}}$

   with respect to the standard bases and show that

   ${\begin{pmatrix}1\\2\\3\end{pmatrix}},\;{\begin{pmatrix}4\\5\\6\end{pmatrix}}$

   are both members of the perp of the nullspace. Prove that ${\mathcal {N}}(f)^{\perp }$ is the span of these two. (Hint. See the third item of Problem 14.)
4. Generalize that to apply to any $f:\mathbb {R} ^{n}\to \mathbb {R} ^{m}$.

This, and related results, is called the Fundamental Theorem of Linear Algebra in (Strang 1993).

Answer

1. The representation of

   ${\begin{pmatrix}v_{1}\\v_{2}\\v_{3}\end{pmatrix}}{\stackrel {f}{\longmapsto }}1v_{1}+2v_{2}+3v_{3}$

   is this.

   ${\rm {Rep}}_{{\mathcal {E}}_{3},{\mathcal {E}}_{1}}(f)={\begin{pmatrix}1&2&3\end{pmatrix}}$

   By the definition of $f$

   ${\mathcal {N}}(f)=\{{\begin{pmatrix}v_{1}\\v_{2}\\v_{3}\end{pmatrix}}\,{\big |}\,1v_{1}+2v_{2}+3v_{3}=0\}=\{{\begin{pmatrix}v_{1}\\v_{2}\\v_{3}\end{pmatrix}}\,{\big |}\,{\begin{pmatrix}1\\2\\3\end{pmatrix}}\cdot {\begin{pmatrix}v_{1}\\v_{2}\\v_{3}\end{pmatrix}}=0\}$

   and this second description exactly says this.

   ${\mathcal {N}}(f)^{\perp }=[\{{\begin{pmatrix}1\\2\\3\end{pmatrix}}\}]$

2. The generalization is that for any $f:\mathbb {R} ^{n}\to \mathbb {R}$ there is a vector ${\vec {h}}$ so that

   ${\begin{pmatrix}v_{1}\\\vdots \\v_{n}\end{pmatrix}}{\stackrel {f}{\longmapsto }}h_{1}v_{1}+\dots +h_{n}v_{n}$

   and ${\vec {h}}\in {\mathcal {N}}(f)^{\perp }$. We can prove this by, as in the prior item, representing $f$ with respect to the standard bases and taking ${\vec {h}}$ to be the column vector gotten by transposing the one row of that matrix representation.
3. Of course,

   ${\rm {Rep}}_{{\mathcal {E}}_{3},{\mathcal {E}}_{2}}(f)={\begin{pmatrix}1&2&3\\4&5&6\end{pmatrix}}$

   and so the nullspace is this set.

   ${\mathcal {N}}(f)\{{\begin{pmatrix}v_{1}\\v_{2}\\v_{3}\end{pmatrix}}\,{\big |}\,\;{\begin{pmatrix}1&2&3\\4&5&6\end{pmatrix}}{\begin{pmatrix}v_{1}\\v_{2}\\v_{3}\end{pmatrix}}={\begin{pmatrix}0\\0\end{pmatrix}}\;\}$

   That description makes clear that

   ${\begin{pmatrix}1\\2\\3\end{pmatrix}},\,{\begin{pmatrix}4\\5\\6\end{pmatrix}}\in {\mathcal {N}}(f)^{\perp }$

   and since ${\mathcal {N}}(f)^{\perp }$ is a subspace of $\mathbb {R} ^{n}$, the span of the two vectors is a subspace of the perp of the nullspace. To see that this containment is an equality, take

   $M=[\{{\begin{pmatrix}1\\2\\3\end{pmatrix}}\}]\qquad N=[\{{\begin{pmatrix}4\\5\\6\end{pmatrix}}\}]$

   in the third item of Problem 14, as suggested in the hint.
4. As above, generalizing from the specific case is easy: for any $f:\mathbb {R} ^{n}\to \mathbb {R} ^{m}$ the matrix $H$ representing the map with respect to the standard bases describes the action

   ${\begin{pmatrix}v_{1}\\\vdots \\v_{n}\end{pmatrix}}{\stackrel {f}{\longmapsto }}{\begin{pmatrix}h_{1,1}v_{1}+h_{1,2}v_{2}+\dots +h_{1,n}v_{n}\\\vdots \\h_{m,1}v_{1}+h_{m,2}v_{2}+\dots +h_{m,n}v_{n}\end{pmatrix}}$

   and the description of the nullspace gives that on transposing the $m$ rows of $H$

   ${\vec {h}}_{1}={\begin{pmatrix}h_{1,1}\\h_{1,2}\\\vdots \\h_{1,n}\end{pmatrix}},\dots {\vec {h}}_{m}={\begin{pmatrix}h_{m,1}\\h_{m,2}\\\vdots \\h_{m,n}\end{pmatrix}}$

   we have ${\mathcal {N}}(f)^{\perp }=[\{{\vec {h}}_{1},\dots ,{\vec {h}}_{m}\}]$. (In (Strang 1993), this space is described as the transpose of the row space of $H$.)

Problem 16

Define a projection to be a linear transformation $t:V\to V$ with the property that repeating the projection does nothing more than does the projection alone: $(t\circ t)\,({\vec {v}})=t({\vec {v}})$ for all ${\vec {v}}\in V$.

1. Show that orthogonal projection onto a line has that property.
2. Show that projection along a subspace has that property.
3. Show that for any such $t$ there is a basis $B=\langle {\vec {\beta }}_{1},\ldots ,{\vec {\beta }}_{n}\rangle$ for $V$ such that

   $t({\vec {\beta }}_{i})={\begin{cases}{\vec {\beta }}_{i}&i=1,2,\dots ,\,r\\{\vec {0}}&i=r+1,r+2,\dots ,\,n\end{cases}}$

   where $r$ is the rank of $t$.
4. Conclude that every projection is a projection along a subspace.
5. Also conclude that every projection has a representation

   ${\rm {Rep}}_{B,B}(t)=\left({\begin{array}{c|c}I&Z\\\hline Z&Z\end{array}}\right)$

   in block partial-identity form.

Answer

1. First note that if a vector ${\vec {v}}$ is already in the line then the orthogonal projection gives ${\vec {v}}$ itself. One way to verify this is to apply the formula for projection onto the line spanned by a vector ${\vec {s}}$, namely $({\vec {v}}\cdot {\vec {s}}/{\vec {s}}\cdot {\vec {s}})\cdot {\vec {s}}$. Taking the line as $\{k\cdot {\vec {v}}\,{\big |}\,k\in \mathbb {R} \}$ (the ${\vec {v}}={\vec {0}}$ case is separate but easy) gives $({\vec {v}}\cdot {\vec {v}}/{\vec {v}}\cdot {\vec {v}})\cdot {\vec {v}}$, which simplifies to ${\vec {v}}$, as required. Now, that answers the question because after once projecting onto the line, the result ${\mbox{proj}}_{\ell }({\vec {v}})$ is in that line. The prior paragraph says that projecting onto the same line again will have no effect.
2. The argument here is similar to the one in the prior item. With $V=M\oplus N$, the projection of ${\vec {v}}={\vec {m}}+{\vec {n}}$ is ${\mbox{proj}}_{M,N}({{\vec {v}}\,})={\vec {m}}$. Now repeating the projection will give ${\mbox{proj}}_{M,N}({\vec {m}})={\vec {m}}$, as required, because the decomposition of a member of $M$ into the sum of a member of $M$ and a member of $N$ is ${\vec {m}}={\vec {m}}+{\vec {0}}$. Thus, projecting twice onto $M$ along $N$ has the same effect as projecting once.
3. As suggested by the prior items, the condition gives that $t$ leaves vectors in the rangespace unchanged, and hints that we should take ${\vec {\beta }}_{1}$, ..., ${\vec {\beta }}_{r}$ to be basis vectors for the range, that is, that we should take the range space of $t$ for $M$ (so that $\dim(M)=r$). As for the complement, we write $N$ for the nullspace of $t$ and we will show that $V=M\oplus N$. To show this, we can show that their intersection is trivial $M\cap N=\{{\vec {0}}\}$ and that they sum to the entire space $M+N=V$. For the first, if a vector ${\vec {m}}$ is in the rangespace then there is a ${\vec {v}}\in V$ with $t({\vec {v}})={\vec {m}}$, and the condition on $t$ gives that $t({\vec {m}})=(t\circ t)\,({\vec {v}})=t({\vec {v}})={\vec {m}}$, while if that same vector is also in the nullspace then $t({\vec {m}})={\vec {0}}$ and so the intersection of the rangespace and nullspace is trivial. For the second, to write an arbitrary ${\vec {v}}$ as the sum of a vector from the rangespace and a vector from the nullspace, the fact that the condition $t({\vec {v}})=t(t({\vec {v}}))$ can be rewritten as $t({\vec {v}}-t({\vec {v}}))={\vec {0}}$ suggests taking ${\vec {v}}=t({\vec {v}})+({\vec {v}}-t({\vec {v}}))$. So we are finished on taking a basis $B=\langle {\vec {\beta }}_{1},\ldots ,{\vec {\beta }}_{n}\rangle$ for $V$ where $\langle {\vec {\beta }}_{1},\ldots ,{\vec {\beta }}_{r}\rangle$ is a basis for the rangespace $M$ and $\langle {\vec {\beta }}_{r+1},\ldots ,{\vec {\beta }}_{n}\rangle$ is a basis for the nullspace $N$.
4. Every projection (as defined in this exercise) is a projection onto its rangespace and along its nullspace.
5. This also follows immediately from the third item.

Problem 17

A square matrix is symmetric if each $i,j$ entry equals the $j,i$ entry (i.e., if the matrix equals its transpose). Show that the projection matrix $A({{A}^{\rm {trans}}}A)^{-1}{{A}^{\rm {trans}}}$ is symmetric. (Strang 1980) Hint. Find properties of transposes by looking in the index under "transpose".

Answer

For any matrix $M$ we have that ${{(M^{-1})}^{\rm {trans}}}=({{M}^{\rm {trans}}})^{-1}$, and for any two matrices $M$, $N$ we have that ${{MN}^{\rm {trans}}}={{N}^{\rm {trans}}}{{M}^{\rm {trans}}}$ (provided, of course, that the inverse and product are defined). Applying these two gives that the matrix equals its transpose.

${{{\bigl (}A({{A}^{\rm {trans}}}A)^{-1}{{A}^{\rm {trans}}}{\bigr )}}^{\rm {trans}}}=({{{A}^{\rm {trans}}}^{\rm {trans}}})({{{\bigl (}({{A}^{\rm {trans}}}A)^{-1}{\bigr )}}^{\rm {trans}}})({{A}^{\rm {trans}}})$

$=({{{A}^{\rm {trans}}}^{\rm {trans}}})({\bigl (}{{({{A}^{\rm {trans}}}A)}^{\rm {trans}}}{\bigr )}^{-1})({{A}^{\rm {trans}}})=A({{A}^{\rm {trans}}}{{{A}^{\rm {trans}}}^{\rm {trans}}})^{-1}{{A}^{\rm {trans}}}=A({{A}^{\rm {trans}}}A)^{-1}{{A}^{\rm {trans}}}$

References

• Strang, Gilbert (1993), "The Fundamental Theorem of Linear Algebra", American Mathematical Monthly, American Mathematical Society: 848–855 {{citation}}: Unknown parameter |month= ignored (help).
• Strang, Gilbert (1980), Linear Algebra and its Applications (2nd ed.), Hartcourt Brace Javanovich