60

DEREK JUNG ADAPTED FROM NOTES BY UCLA PROFESSOR PETER PETERSEN

6. Inner product spaces? ... More like winner product spaces! Actual conversation with Hadrian Quan... Me: Hey, I thought of a title for my chapter about inner product spaces. Hadrian: Oh, what is that? Me: Oh, it’s a space with an inner product. I begin this chapter by briefly describing the origin of the study of inner product spaces... Very few know that the theory of inner product spaces originated in Hamburg, Germany during the late 19th century. While the sandwich idea of a hamburger did not come about until later, cooking meat patties with spices became very popular in the large German town. Hamburg cows were being killed at an unprecedented rate as a result of the increased rise in demand. Butchers would typically only use certain parts of the cow, then discard materials like the intestines and vital organs. Consequently, the sewage system became overwhelmed and waste began flowing into the neighboring Baltic Sea. This resulted in an algal bloom, and the sea became so bright, so vivid that the color became what we now call Pythagreen. The city desperately needed to do something because the increased algae production was decimating the native sea life. How one wanted to solve this issue, however, quickly became one of the most polarizing identities. Some proposed that they should stop selling hamburger patties, but there was ardent backlash due to its popularity. Others argued that the environmental issue would figure itself out after the sea acclimated to the change, but the city’s scientists dismissed this as illogical. Finally, one group decided that it would be best to try to retrofit the city’s system of aquaducts to become an ultrafilter. They figured out a way to slow down the flow of sewage to a rate at which the Baltic Sea could safely handle. While this group was once closed under addition, they quickly became popular because their simple signup sheet consisted of two equally spaced lines. This symmetric bilinear form allowed them to easily spread their message, which helped them be successful. This group was named Innards Pro-ducts pace, and they eventually adapted their ideas to invent what we know as inner product spaces. We will assume F = R or F = C throughout this chapter. Which one F is (or if it matters) should hopefully be clear from context. 6.1. Inner products. We would like to have more structure on our vector spaces. Recall that we can only consider finite sums of vectors as we have no concept of distance. In this section, we will consider vector spaces for which we can consider lengths of vectors and angles between pairs of vectors. Recall given x + iy ∈ C, we define the conjugate a + ib := a − ib. We view R ⊂ C by identifying R with the real axis of C. In particular, a = a ¯ for a ∈ R. Definition 6.1.1. Let V be a vector space over F. An inner product is a map h·, ·i : V ×V → F satisfying the following three properties: For all x, y ∈ V , (1) (Non-degeneracy) ||x||2 := hx, xi > 0 if x 6= 0. (2) (Complex-symmetry) hx, yi = hy, xi. (3) (Linearity in the first component) The map z 7→ hz, yi : V → F is linear, i.e., hαz1 + z2 , yi = αhz1 , yi + hz2 , yi for all z1 , z2 ∈ V, α ∈ F.

SWILA NOTES

61

We call (V, h·, ·i) a real inner product space (complex inner product space) if F = R (if F = C). p For x ∈ V , we define ||x|| = hx, xi ≥ 0 to be the norm of x. We will simply say inner product space and drop the mention of the scalars if the discussion works for both R and C. Remark 6.1.2. In the case F = R, note complex-symmetry is the same as symmetry since a = a ¯ for all a ∈ R. Note that ||0|| = 0 from linearity in the first component. Finally, hx, αyi = α ¯ hx, yi and ||αx||2 = |α|2 ||x||2 for all a ∈ C, x, y ∈ V . Example 6.1.3. Rn becomes a real inner product space with the inner product defined by n X h(x1 , . . . , xn ), (y1 , . . . , yn )i := xj yj = ( x1 · · · xn )( y1 · · · yn )t . j=1

Observe that given two nonzero vectors x, y, the angle θ between x and y is given by hx, yi . θ= ||x|| · ||y|| Moreover, two vectors x,y are orthogonal if and only if hx, yi = 0. Example 6.1.4. Cn is a complex inner product space with the complex inner product defined by n X h(w1 , . . . , wn ), (z1 , . . . , zn )i := wj zj = (w1 , . . . , wn )(z1 , . . . , zn )∗ . j=1

Example 6.1.5. (Inner product on square matrices) Given a matrix A = (aij ) ∈ Mn (C), recall that we define the trace of A as n X tr(A) := aii . i=1

We can define an inner product on the matrices Mn (C) by hA, Bi := tr(B ∗ A). It is clear that this inner product is linear in the first component. By Lemma 2.5.2, hA, Bi = tr(B ∗ A) = tr((B ∗ A)∗ ) = tr(A∗ B) = hB, Ai. It remains to show tr(A∗ A) > 0 for all 0 6= A ∈ Mn (C). It’s nontrivial to show this at this juncture, so we will prove this after proving the Spectral Theorem for self-adjoint operators (see Remark 7.4.8). The same proof shows that the trace is also an inner product on Mn (R). Example 6.1.6. The space of complex-valued continuous functions C([0, 1]) becomes an inner product space when endowed with the inner product Z 1 hf, gi := f (x)g(x) dx. 0

However, this is not an inner product on the space of integrable functions L1 ([0, 1]) (why?). Example 6.1.7. The vector space l2 (read ”little el two”) of square-summable complex sequences becomes an inner product space with the inner product ∞ X h(aj ), (bj )i := aj b j . j=1

62

DEREK JUNG ADAPTED FROM NOTES BY UCLA PROFESSOR PETER PETERSEN

This is one of the most important examples of inner product spaces because it forms a Hilbert space. Unfortunately, we will not have enough time to talk about Hilbert spaces in this workshop. Fortunately, all finite-dimensional inner product spaces are Hilbert spaces. A sometimes useful property of inner product spaces is the parallelogram law, which follows from linearity. Proposition 6.1.8. (Parallelogram law) Let V be an inner product space. Then for all x, y ∈ V , ||x + y||2 + ||x − y||2 = 2(||x||2 + ||y||2 ). The two most important properties of inner product spaces are the Cauchy-Schwarz inequality and the triangle inequality. We will prove those here. Proposition 6.1.9. (Cauchy-Schwarz inequality) Let V be an inner product space. For all x, y ∈ V , |hx, yi| ≤ ||x|| · ||y||. Proof. This is the easiest to remember proof I know. We will assume F = C as the result for F = R follows (or by the same proof). We may assume y is nonzero, or hx, yi = 0 by linearity. For all z ∈ C, 0 ≤ ||x + zy||2 = hx + zy, x + zyi = ||x||2 + hzy, xi + hx, zyi + ||zy||2 (complex-symmetry + linearity)

= ||x||2 + 2Re(zhy, xi) + |z|2 ||y||2 .

As hy, xi ∈ C, we may write hy, xi = |hx, yi|eiθ ,

r ≥ 0, θ ∈ R.

Write z = se−iθ for some s ≥ 0. Then 0 ≤ ||x||2 + 2s|hx, yi| + s2 ||y||2 . The right side is the equation of a U -shaped parabola, which has its minimum at |hx, yi| s := − . ||y||2 At this value of s, |hx, yi|2 0 ≤ ||x||2 + 2s|hx, yi| + s2 ||y||2 = ||x||2 − . ||y||2 Rearranging terms, the Cauchy-Schwarz inequality follows.



Remark 6.1.10. I have a few remarks about the above proof. We will often assume F = C when working with inner product spaces. The proof in the case F = R often follows since we view R ⊆ C after identifying R with the real axis of C. Using bigger words, we can often reduce to the case F = C as R is isometrically isomorphic to a subspace of C. On another note, observe that we used the assumption y 6= 0 to assure that we weren’t dividing by zero in defining s. For the proof, we could have assumed x 6= 0 as well, but I didn’t want to confuse readers in figuring out where we used that x is nonzero. Typically in math proofs, it is very useful to simplify things by making certain reducing assumptions. For example, by using the polar decomposition of complex numbers, we reduced our proof to the case we had a real inner product space.

SWILA NOTES

63

We can now prove the triangle inequality for inner product spaces. Corollary 6.1.11. (Triangle inequality) Let V be an inner product space. For all x, y ∈ V , ||x + y|| ≤ ||x|| + ||y||. Proof. For all x, y ∈ V , ||x + y||2 = ||x||2 + 2Rehx, yi + ||y||2 ≤ ||x||2 + 2|hx, yi| + ||y||2 ≤ ||x||2 + 2||x|| · ||y|| + ||y||2

(Cauchy-Schwarz inequality)

= (||x|| + ||y||)2 .  Remark 6.1.12. The triangle inequality implies that an inner product space is actually a normed space endowed with the norm || · ||.Unfortunately, we will not have time to discuss normed spaces. 6.2. Angles and orthogonality in inner product spaces. In high school geometry, we learned that we can figure out the lengths of sides of triangles using angles. For example, there are the Law of Sines and the Law of Cosines. A powerful tool of inner product spaces (which differentiates them from more general normed spaces) is that we can discuss angles. Basically, given two sides of a triangle, we can figure out the third side. This is a very useful tool when discussing limits and estimating. Definition 6.2.1. Let V be an inner product space. We define the angle between two nonzero vectors x, y ∈ V to be hx, yi . ||x|| · ||y|| We say that vectors x, y ∈ V (possibly zero) are othogonal if hx, yi = 0. We say that a subset S ⊆ V is orthogonal if any two distinct vectors in S are orthogonal. Remark 6.2.2. Observe that the zero vector is orthogonal to every other vector in an inner product space. In terms of equivalence relations, note that declaring x∼y

⇐⇒

“x is orthogonal to y”

satisfies symmetry, but neither reflexivity nor transitivity in general. Definition 6.2.3. The projection of a vector x in the direction of a nonzero vector y is given by   y y projy (x) = x, . ||y|| ||y|| Observe that two vectors x, y in an inner product space are orthogonal if and only if ||x + y||2 = ||x||2 + ||y||2 . Many results hold for real and complex inner product spaces; the following frequently used theorem is an example of such.

64

DEREK JUNG ADAPTED FROM NOTES BY UCLA PROFESSOR PETER PETERSEN

Theorem 6.2.4. (Pythagorean theorem) Let V be an inner product space and suppose x1 , . . . , xn ∈ V are orthogonal. Then ||

n X k=1

xk ||2 =

n X

||xk ||2 .

k=1

Proof. Induction on k, jk induct on n. You always induct on n. ALWAYS. -DJung



6.3. Orthonormal bases for finite-dimensional vector spaces. Throughout this section, we will assume V is a finite-dimensional inner product space. We will show in this section that each of these spaces is “pretty much” the same as some inner product space we already know and love. Recall the definition of the Kronecker delta  1, if i = j δij := 0, if i 6= j. Definition 6.3.1. Let V be a finite-dimensional inner product space. We say a collection of vectors {v1 , . . . , vn } ⊆ V is orthonormal if hvi , vj i = δij for all i, j. We say a basis of V is an orthonormal basis if it is orthonormal. Example 6.3.2. An orthonormal basis for Cn is the standard basis S. If n ≥ 3, another orthonormal basis for Cn would be p p p p {e3 , e4 , . . . , en } ∪ {( 1/2, 1/2, 0, . . . 0), ( 1/2, − 1/2, 0, . . . , 0)}. In particular, orthonormal bases are not unique in general. Remark 6.3.3. It becomes a bit more difficult to talk about orthonormal bases for infinitedimensional vector spaces. One could use an analogous definition, allowing our bases to be infinite. However, a more useful definition is allowing for “linear combinations involving infinite series.” We are allowed to do this because the inner product allows us to talk about limits of linear combinations. Proposition 6.3.4. (The Gram-Schmidt process) Given a collection of vectors {w1 , . . . , wm }, at least one nonzero, there is an orthonormal set of vectors {v1 , . . . , vn }, n ≤ m, such that span{w1 , . . . , wm } = span{v1 , . . . , vn }. The proof is recursive; it involves projecting a new vector onto the span of the previously chosen vectors and then taking a difference. A proof can be found in Petersen’s notes [6] on pages 168-169. We obtain the existence of orthonormal bases as a consequence. Corollary 6.3.5. (Existence of orthonormal bases in finite-dimensions) Every finite-dimensional inner product space has an orthonormal basis. Proof. Apply the Gram-Schmidt process to a basis for the space.



Random Thought 6.3.6. This adorable little grandma, Mrs. Lee, discovered that you could save short videos in the form of gifs. Hearing this, her grandson Frank suggested she go by the river to record some of the beavers at work. She responded, “Frank Lee, my dear, I don’t gif a dam.”

SWILA NOTES

65

Proposition 6.3.7. Suppose V is an inner product space with an orthonormal basis {v1 , . . . , vn }. Then each x ∈ V can be (uniquely) expanded as x = hx, v1 iv1 + · · · + hx, vn ivn . Proof. Fix x ∈ V . As {v1 , . . . , vn } is a basis for V , we may write x = a1 v1 + · · · + an vn . Using the properties of the inner product, we can then calculate n X hx, vi i = aj δij = ai . j=1

The uniqueness just comes from recalling that given any basis (not necessarily orthonormal), every element can be uniquely written as a linear combination of the basis vectors (see Section 1.4).  The next corollary follows immediately. Corollary 6.3.8. Let T : V → V be a linear operator and {v1 , . . . , vn } an orthonormal basis for V . Then for all x ∈ V , T (x) = hT (x), v1 iv1 + · · · + hT (x), vn ivn . Recall that given metric spaces X, Y , an isometry between X and Y is a function f : X → Y satisfying dX (a, b) = dY (f (a), f (b)) for all a, b ∈ X. This motivates the following definition. Definition 6.3.9. Let V, W be inner product spaces. An isomorphism T : V → W is called an isometric isomorphism if hT x, T yiW = hx, yiV for all x, y ∈ V . We say vector spaces V and W are isometrically isomorphic if there exists an isometric isomorphism between them. Remark 6.3.10. (Aside about my own studies) When dealing with general vector spaces, we were interested in analyzing isomorphisms. In this section, when the inner product endows us with more structure, we will be interested in isometric isomorphisms. In Differentiable Manifolds (Math 518), one is interested in smooth bijections with smooth inverse since one has a “smooth structure” on the manifold. For example, the plane R2 seems like a pretty nice place to travel on (or a Stars Wars-themed mode of transportation). In metric spaces, one is interested in studying isometries. However, sometimes it’s a bit much to assume that there any many isometries between two metric spaces. Thus, we often settle to study the larger space of Lipschitz functions. We say a function f : X → Y is L-Lipschitz if dY (f (a), f (b)) ≤ L · dX (a, b)

for all a, b ∈ X.

Note isometries are 1-Lipschitz while differentiable functions f : R → R with unbounded derivative are not L-Lipschitz for any constant L. I study Sobolev spaces on metric spaces, which would be interesting to those who really enjoy measure theory and metric spaces. For a brief introduction, say M and N are nice enough spaces (for example, M = Rm and N = Rn ) so that we can make sense of integrals and gradients of functions. We say f : M → N is a Sobolev function if f and its gradient Of are both integrable. I am interested in studying whether Lipschitz functions are dense in certain Sobolev spaces (after defining some notion of distance in Sobolev spaces). It turns out that this problem has a lot to do with algebraic topology, differential geometry, and

66

DEREK JUNG ADAPTED FROM NOTES BY UCLA PROFESSOR PETER PETERSEN

geometric measure theory. More specifically, the density of Lipschitz functions often depends on Lipschitz homotopy groups (which are the same as normal homotopy groups, except one requires the functions and homotopies to be Lipschitz as opposed to just continuous) and the smooth structures of the domain and target space. A related problem is studying whether smooth functions are dense in a Sobolev space; this is in some cases related to whether certain homotopy groups of the target space are trivial and whether one can extend functions from skeletons of the CW complex to the whole space. See various works of Professor Piotr Haljasz and Professor Jeremy Tyson for more information on density of certain spaces in Sobolev spaces. See PDE’s for an introduction to Sobolev spaces on Rn . See Hajlasz’s paper “Sobolev spaces on metric-measure spaces” for an introduction to defining Sobolev spaces between metric spaces when one doesn’t necessarily have a Euclidean structure (one can use path families). We leave the following lemma as a computation to the reader. Lemma 6.3.11. (Polarization identities) Fix an inner product space V and x, y ∈ V . If F = R,
1 hx, yi = ||x + y||2 − ||x − y||2 . 4 If F = C, 4

hx, yi =


1X j 1 i ||x + ij y||2 = ||x + y||2 + i||x + iy||2 − ||x − y||2 − i||x − iy||2 . 4 j=1 4

As one can naturally define a metric on inner product spaces using the inner product, the following proposition gives that isometric isomorphisms are indeed isometries. Proposition 6.3.12. Let T : V → W be an isomorphism between F-inner product spaces. Then the following are equivalent: (1) T is an isometric isomorphism; (2) ||T x||W = ||x||V for all x ∈ V ; (3) ||T x − T y||W = ||x − y||V for all x, y ∈ V . Proof. (2) ⇔ (3) is clear by linearity of T and letting y = 0. (1) ⇒ (2) follows by letting x = y. Finally, (2) ⇒ (1) follows from the Polarization identities by writing things out.  Recall we view Fn as an inner product space with the inner product defined in Example 6.1.3 or Example 6.1.4. Proposition 6.3.13. Let V be an inner product space of dimension n. Then V is isometrically isomorphic to Fn . Proof. By Corollary 6.3.5, there exists an orthonormal basis {v1 , . . . , vn } of V . Define the linear transformation T : V → Fn by x 7→ (hx, v1 i, . . . , hx, vn i). By Lemma 6.3.7, this is an isomorphism with inverse (a1 , . . . , an ) 7→ a1 v1 + · · · + an vn .

SWILA NOTES

67

Moreover, for all x ∈ V , ||x||2 = ||hx, v1 iv1 + · · · + hx, vn ivn ||2

(by Lemma 6.3.7) (by the Pythagorean Theorem)

= ||hx, v1 iv1 ||2 + · · · + ||hx, vn ivn ||2 = |hx, v1 i|2 + · · · + |hx, vn i|2 = ||T x||2 .

By the previous lemma, it follows that V is isometrically isomorphic to Fn .



6.4. Orthogonal complements and projections. Recall in Section 3.2 we defined complements of subspaces and projections. In this section, we will take a more refined look at them in the context of inner product spaces. This perspective wasn’t available to us before because we didn’t have the power of an inner product on our vector spaces to be able to talk about orthogonality. Throughout this section, we will assume V is a finite-dimensional inner product space and F = R or C. Definition 6.4.1. Given a subspace M ⊆ V , fix an orthonormal basis {v1 , . . . , vm } for M . We define the orthogonal projection onto M to be the linear operator projM : V → V defined by projM (x) = hx, v1 iv1 + · · · + hx, vm ivm . Remark 6.4.2. It’s a fact that the projection onto M is well-defined, i.e., independent of the basis chosen for M . Definition 6.4.3. Fix a subspace M ⊆ V . We define the orthogonal complement of M to be M ⊥ := {x ∈ V : hx, yi = 0 for all y ∈ M }. Proposition 6.4.4. Fix a subspace M ⊆ V . Then ker projM = M ⊥ and V = M ⊕ M ⊥ . In particular, (M ⊥ )⊥ = M and dim(V ) = dim(M ) + dim(M ⊥ ). Proof. Note V = M ⊕ M ⊥ from Proposition 3.2.8. To show ker projM ⊆ M ⊥ , suppose projM (x) = 0 for some x ∈ V . We may decompose x = xM + xM ⊥ for xM ∈ M and xM ⊥ ∈ M ⊥ . Note projM (xM ) = xM since proj2M = projM and Im(projM ) = M . Hence, 0 = projM (x) = projM (xM ) + projM (xM ⊥ ) = xM + projM (xM ⊥ ). Hence, xM = −projM (xM ⊥ ) ∈ M ∩ M ⊥ = {0}. ∈ M ⊥ . The other direction is clear since for x ∈ M ⊥ , x = 0 + x with

In particular, x = xM ⊥ 0 ∈ M. For any x ∈ M , note x ∈ (M ⊥ )⊥ . This implies M ⊆ (M ⊥ )⊥ . On the other hand, since M ⊥ ⊕ (M ⊥ )⊥ = M ⊕ M ⊥ , dim((M ⊥ )⊥ ) = dim(V ) − dim(M ⊥ ) = dim(M ). It follows that M = (M ⊥ )⊥ . Random Thought 6.4.5. Have you heard the one about the towel and the desert? Actually nevermind... It’s pretty dry humor.