Linear Algebra/Projection Onto a Subspace

Linear Algebra
 ← Gram-Schmidt Orthogonalization Projection Onto a Subspace Topic: Line of Best Fit → 

This subsection, like the others in this section, is optional. It also requires material from the optional earlier subsection on Combining Subspaces.

The prior subsections project a vector onto a line by decomposing it into two parts: the part in the line and the rest . To generalize projection to arbitrary subspaces, we follow this idea.

Definition 3.1

For any direct sum and any , the projection of onto along is

where with .

This definition doesn't involve a sense of "orthogonal" so we can apply it to spaces other than subspaces of an . (Definitions of orthogonality for other spaces are perfectly possible, but we haven't seen any in this book.)

Example 3.2

The space of matrices is the direct sum of these two.

To project

onto along , we first fix bases for the two subspaces.

The concatenation of these

is a basis for the entire space, because the space is the direct sum, so we can use it to represent .

Now the projection of onto along is found by keeping the part of this sum and dropping the part.

Example 3.3

Both subscripts on are significant. The first subscript matters because the result of the projection is an , and changing this subspace would change the possible results. For an example showing that the second subscript matters, fix this plane subspace of and its basis

and compare the projections along two different subspaces.

(Verification that and is routine.) We will check that these projections are different by checking that they have different effects on this vector.

For the first one we find a basis for

and represent with respect to the concatenation .

The projection of onto along is found by keeping the part and dropping the part.

For the other subspace , this basis is natural.

Representing with respect to the concatenation

and then keeping only the part gives this.

Therefore projection along different subspaces may yield different results.

These pictures compare the two maps. Both show that the projection is indeed "onto" the plane and "along" the line.

Linalg projection onto plane.png Linalg projection onto plane 2.png

Notice that the projection along is not orthogonal— there are members of the plane that are not orthogonal to the dotted line. But the projection along is orthogonal.

A natural question is: what is the relationship between the projection operation defined above, and the operation of orthogonal projection onto a line? The second picture above suggests the answer— orthogonal projection onto a line is a special case of the projection defined above; it is just projection along a subspace perpendicular to the line.

Linalg projection onto line.png

In addition to pointing out that projection along a subspace is a generalization, this scheme shows how to define orthogonal projection onto any subspace of , of any dimension.

Definition 3.4

The orthogonal complementof a subspace of is

(read " perp"). The orthogonal projection of a vector is its projection onto along .

Example 3.5

In , to find the orthogonal complement of the plane

we start with a basis for .

Any perpendicular to every vector in is perpendicular to every vector in the span of (the proof of this assertion is Problem 10). Therefore, the subspace consists of the vectors that satisfy these two conditions.

We can express those conditions more compactly as a linear system.

We are thus left with finding the nullspace of the map represented by the matrix, that is, with calculating the solution set of a homogeneous linear system.

Example 3.6

Where is the -plane subspace of , what is ? A common first reaction is that is the -plane, but that's not right. Some vectors from the -plane are not perpendicular to every vector in the -plane.

Linalg xyperp not yz.png

Instead is the -axis, since proceeding as in the prior example and taking the natural basis for the -plane gives this.

The two examples that we've seen since Definition 3.4 illustrate the first sentence in that definition. The next result justifies the second sentence.

Lemma 3.7

Let be a subspace of . The orthogonal complement of is also a subspace. The space is the direct sum of the two . And, for any , the vector is perpendicular to every vector in .

Proof

First, the orthogonal complement is a subspace of because, as noted in the prior two examples, it is a nullspace.

Next, we can start with any basis for and expand it to a basis

for the entire space. Apply the Gram-Schmidt process to get an orthogonal basis for . This is the concatenation of two bases (with the same number of members as ) and . The first is a basis for , so if we show that the second is a basis for then we will have that the entire space is the direct sum of the two subspaces.

Problem 9 from the prior subsection proves this about any orthogonal basis: each vector in the space is the sum of its orthogonal projections onto the lines spanned by the basis vectors.

To check this, represent the vector , apply to both sides , and solve to get , as desired.

Since obviously any member of the span of is orthogonal to any vector in , to show that this is a basis for we need only show the other containment— that any is in the span of this basis. The prior paragraph does this. On projections onto basis vectors from , any gives and therefore () gives that is a linear combination of . Thus this is a basis for and is the direct sum of the two.

The final sentence is proved in much the same way. Write . Then is gotten by keeping only the part and dropping the part . Therefore consists of a linear combination of elements of and so is perpendicular to every vector in .

We can find the orthogonal projection onto a subspace by following the steps of the proof, but the next result gives a convienent formula.

Theorem 3.8

Let be a vector in and let be a subspace of with basis . If is the matrix whose columns are the 's then where the coefficients are the entries of the vector . That is, .

Proof

The vector is a member of and so it is a linear combination of basis vectors . Since 's columns are the 's, that can be expressed as: there is a such that (this is expressed compactly with matrix multiplication as in Example 3.5 and 3.6). Because is perpendicular to each member of the basis, we have this (again, expressed compactly).

Solving for (showing that is invertible is an exercise)

gives the formula for the projection matrix as .

Example 3.9

To orthogonally project this vector onto this subspace

first make a matrix whose columns are a basis for the subspace

and then compute.

With the matrix, calculating the orthogonal projection of any vector onto is easy.

ExercisesEdit

This exercise is recommended for all readers.
Problem 1

Project the vectors onto   along  .

  1.  
  2.  
  3.  
This exercise is recommended for all readers.
Problem 2

Find  .

  1.  
  2.  
  3.  
  4.  
  5.  
  6.  
  7.  
Problem 3

This subsection shows how to project orthogonally in two ways, the method of Example 3.2 and 3.3, and the method of Theorem 3.8. To compare them, consider the plane   specified by   in  .

  1. Find a basis for  .
  2. Find   and a basis for  .
  3. Represent this vector with respect to the concatenation of the two bases from the prior item.
     
  4. Find the orthogonal projection of   onto   by keeping only the   part from the prior item.
  5. Check that against the result from applying Theorem 3.8.
This exercise is recommended for all readers.
Problem 4

We have three ways to find the orthogonal projection of a vector onto a line, the Definition 1.1 way from the first subsection of this section, the Example 3.2 and 3.3 way of representing the vector with respect to a basis for the space and then keeping the   part, and the way of Theorem 3.8. For these cases, do all three ways.

  1.  
  2.  
Problem 5

Check that the operation of Definition 3.1 is well-defined. That is, in Example 3.2 and 3.3, doesn't the answer depend on the choice of bases?

Problem 6

What is the orthogonal projection onto the trivial subspace?

Problem 7

What is the projection of   onto   along   if  ?

Problem 8

Show that if   is a subspace with orthonormal basis   then the orthogonal projection of   onto   is this.

 
This exercise is recommended for all readers.
Problem 9

Prove that the map   is the projection onto   along   if and only if the map   is the projection onto   along  . (Recall the definition of the difference of two maps:  .)

This exercise is recommended for all readers.
Problem 10

Show that if a vector is perpendicular to every vector in a set then it is perpendicular to every vector in the span of that set.

Problem 11

True or false: the intersection of a subspace and its orthogonal complement is trivial.

Problem 12

Show that the dimensions of orthogonal complements add to the dimension of the entire space.

This exercise is recommended for all readers.
Problem 13

Suppose that   are such that for all complements  , the projections of   and   onto   along   are equal. Must   equal  ? (If so, what if we relax the condition to: all orthogonal projections of the two are equal?)

This exercise is recommended for all readers.
Problem 14

Let   be subspaces of  . The perp operator acts on subspaces; we can ask how it interacts with other such operations.

  1. Show that two perps cancel:  .
  2. Prove that   implies that  .
  3. Show that  .
This exercise is recommended for all readers.
Problem 15

The material in this subsection allows us to express a geometric relationship that we have not yet seen between the rangespace and the nullspace of a linear map.

  1. Represent   given by
     
    with respect to the standard bases and show that
     
    is a member of the perp of the nullspace. Prove that   is equal to the span of this vector.
  2. Generalize that to apply to any  .
  3. Represent  
     
    with respect to the standard bases and show that
     
    are both members of the perp of the nullspace. Prove that   is the span of these two. (Hint. See the third item of Problem 14.)
  4. Generalize that to apply to any  .

This, and related results, is called the Fundamental Theorem of Linear Algebra in (Strang 1993).

Problem 16

Define a projection to be a linear transformation   with the property that repeating the projection does nothing more than does the projection alone:   for all  .

  1. Show that orthogonal projection onto a line has that property.
  2. Show that projection along a subspace has that property.
  3. Show that for any such   there is a basis   for   such that
     
    where   is the rank of  .
  4. Conclude that every projection is a projection along a subspace.
  5. Also conclude that every projection has a representation
     
    in block partial-identity form.
Problem 17

A square matrix is symmetric if each   entry equals the   entry (i.e., if the matrix equals its transpose). Show that the projection matrix   is symmetric. (Strang 1980) Hint. Find properties of transposes by looking in the index under "transpose".

Solutions

ReferencesEdit

  • Strang, Gilbert (Nov. 1993), "The Fundamental Theorem of Linear Algebra", American Mathematical Monthly (American Mathematical Society): 848-855 .
  • Strang, Gilbert (1980), Linear Algebra and its Applications (2nd ed.), Hartcourt Brace Javanovich 
Linear Algebra
 ← Gram-Schmidt Orthogonalization Projection Onto a Subspace Topic: Line of Best Fit →