How to repair project onto ? Dont change the procedure. Fix the - PowerPoint PPT Presentation

How to repair project onto ? Don’t change the procedure. Fix the spec. Require that vlist consists of mutually orthogonal vectors: the i th vector in the list is orthogonal to the j th vector in the list for every i 6 = j .

The return of project onto I input: a vector b , a list vlist [ v 1 , . . . , v n ] of mutually orthogonal vectors I output: the projection of b onto the space spanned by v 1 , . . . , v n def project_onto(b, vlist): return sum([project_along(b, v) for v in vlist]) Let ˆ b be the result. Need to prove I ˆ b lies in Span { v 1 , . . . , v n } , and I b � ˆ b is orthogonal to Span { v 1 , . . . , v n } Su ffi ces to show that b � ˆ b is orthogonal to each of v 1 , . . . , v n for then it is orthogonal to every linear combination

Proving the correctness of project onto def project onto(b, vlist): return sum([project along(b, v) for v in vlist]) Let ˆ b be the result. Need to prove 1. ˆ b lies in Span { v 1 , . . . , v n } , and 2. b � ˆ b is orthogonal to Span { v 1 , . . . , v n } Su ffi ces to show that b � ˆ b is orthogonal to each of v 1 , . . . , v n for then it is orthogonal to every linear combination (1) By correctness of project along ( b , v ), the result is a scalar multiple of v for each vector v in vlist . Thus ˆ b = σ 1 v 1 + . . . σ n v n where σ 1 , . . . , σ n are the scalars. This is a linear combination of v 1 , . . . , v n , so ˆ b belongs to Span { v 1 , . . . , v n } .

Proving the correctness of project onto Need to prove 1. ˆ b lies in Span { v 1 , . . . , v n } , and 2. b � ˆ b is orthogonal to Span { v 1 , . . . , v n } Su ffi ces to show that b � ˆ b is orthogonal to each of v 1 , . . . , v n for then it is orthogonal to every linear combination (2) For i = 1 , 2 , . . . , n , D E D ˆ E b � ˆ b , v i = h b , v i i � b , v i h b , v i i � h σ 1 v 1 � σ 2 v 2 + · · · � σ i v i � · · · � σ i h v i , v i i � · · · � σ n v n , v i i = = h b , v i i � σ 1 h v 1 , v i i � σ 2 h v 2 , v i i � · · · � h v n , v i i h b , v i i � 0 � 0 � · · · � σ i h v i , v i i � · · · � 0 = = h b , v i i � σ i h v i , v i i b || v i + b ⊥ , v i , v i D E � σ i h v i , v i i = b || v i , v i b ⊥ v i , v i D E D E = + � σ i h v i , v i i = h σ i v i , v i i + 0 � σ i h v i , v i i = 0

A new subroutine: project orthogonal(b, vlist) We have proved that project onto(b, vlist) satisfies its spec: I input: vector b , list vlist of mutually orthogonal vectors I output: projection of b onto the span of vectors in vlist Use this to build a subroutine project orthogonal(b, vlist) with spec: I input: vector b , list vlist of mutually orthogonal vectors I output: projection of b orthogonal to the span of vectors in vlist def project orthogonal(b, vlist): return b - project onto(b, vlist)

Building an orthogonal set of generators Original stated goal: Find the projection of b onto the space V spanned by arbitrary vectors v 1 , . . . , v n . So far we know how to find the projection of b onto the space spanned by mutually orthogonal vectors. This would su ffi ce if we had a procedure that, given arbitrary vectors v 1 , . . . , v n , computed mutually orthogonal vectors v ∗ 1 , . . . , v ∗ n that span the same space. We consider a new problem: orthogonalization : I input: A list [ v 1 , . . . , v n ] of vectors over the reals I output: A list of mutually orthogonal vectors v ∗ 1 , . . . , v ∗ n such that Span { v ∗ 1 , . . . , v ∗ n } = Span { v 1 , . . . , v n } How can we solve this problem?

The orthogonalize procedure Idea: Use project orthogonal iteratively to make a longer and longer list of mutually orthogonal vectors. I First consider v 1 . Define v ∗ 1 := v 1 since the set { v ∗ 1 } is trivially a set of mutually orthogonal vectors. I Next, define v ∗ 2 to be the projection of v 2 orthogonal to v ∗ 1 . I Now { v ∗ 1 , v ∗ 2 } is a set of mutually orthogonal vectors. I Next, define v ∗ 3 to be the projection of v 3 orthogonal to v ∗ 1 and v ∗ 2 , so { v ∗ 1 , v ∗ 2 , v ∗ 3 } is a set of mutually orthogonal vectors.... In each step, we use project orthogonal to find the next orthogonal vector. In the i th iteration, we project v i orthogonal to v ∗ 1 , . . . , v ∗ i − 1 to find v ∗ i . def orthogonalize(vlist): vstarlist = [] for v in vlist: vstarlist.append(project_orthogonal(v, vstarlist)) return vstarlist

Correctness of the orthogonalize procedure, Part I def orthogonalize(vlist): vstarlist = [] for v in vlist: vstarlist.append(project_orthogonal(v, vstarlist)) return vstarlist Lemma: Throughout the execution of orthogonalize , the vectors in vstarlist are mu- tually orthogonal. In particular, the list vstarlist at the end of the execution, which is the list returned, consists of mutually orthogonal vectors. Proof: by induction, using the fact that each vector added to vstarlist is orthogonal to all the vectors already in the list. QED

Example of orthogonalize Example: When orthogonalize is called on a vlist consisting of vectors v 1 = [2 , 0 , 0] , v 2 = [1 , 2 , 2] , v 3 = [1 , 0 , 2] it returns the list vstarlist consisting of v ∗ 1 = [2 , 0 , 0] , v ∗ 2 = [0 , 2 , 2] , v ∗ 3 = [0 , � 1 , 1] (1) In the first iteration, when v is v 1 , vstarlist is empty, so the first vector v ∗ 1 added to vstarlist is v 1 itself. (2) In the second iteration, when v is v 2 , vstarlist consists only of v ∗ 1 . The projection of v 2 v 2 � h v 2 , v ∗ orthogonal to v ∗ 1 i 1 = [1 , 2 , 2] � 2 1 is 1 i v ∗ 4[2 , 0 , 0] = [0 , 2 , 2] h v ∗ 1 , v ∗ so v ∗ 2 = [0 , 2 , 2] is added to vstarlist . (3) In the third iteration, when v is v 3 , vstarlist consists of v ∗ 1 and v ∗ 2 . The projection of v 3 orthogonal to v ∗ 1 is [0 , 0 , 2], and the projection of [0 , 0 , 2] orthogonal to v ∗ 2 is [0 , 0 , 2] � 1 2[0 , 2 , 2] = [0 , � 1 , 1] so v ∗ 3 = [0 , � 1 , 1] is added to vstarlist

Correctness of the orthogonalize procedure, Part II Lemma: Consider orthogonalize applied to an n -element list [ v 1 , . . . , v n ]. After i iterations of the algorithm, Span vstarlist = Span { v 1 , . . . , v i } . Proof: by induction on i . The case i = 0 is trivial. After i � 1 iterations, vstarlist consists of vectors v ∗ 1 , . . . , v ∗ i − 1 . Assume the lemma holds at this point. This means that Span { v ∗ 1 , . . . , v ∗ i − 1 } = Span { v 1 , . . . , v i − 1 } By adding the vector v i to sets on both sides, we obtain Span { v ∗ 1 , . . . , v ∗ i − 1 , v i } = Span { v 1 , . . . , v i − 1 , v i } ... It therefore remains only to show that Span { v ∗ 1 , . . . , v ∗ i − 1 , v ∗ i } = Span { v ∗ 1 , . . . , v ∗ i − 1 , v i } . The i th iteration computes v ∗ i using project orthogonal ( v i , [ v ∗ 1 , . . . , v ∗ i − 1 ]). There are scalars α i 1 , α i 2 , . . . , α i , i − 1 such that v i = α 1 i v ∗ 1 + · · · + α i − 1 , i v ∗ i − 1 + v ∗ i This equation shows that any linear combination of

Correctness of the orthogonalize procedure, Part II Lemma: Consider orthogonalize applied to an n -element list [ v 1 , . . . , v n ]. After i iterations of the algorithm, Span vstarlist = Span { v 1 , . . . , v i } . Proof: by induction on i . ... It therefore remains only to show that Span { v ∗ 1 , . . . , v ∗ i − 1 , v ∗ i } = Span { v ∗ 1 , . . . , v ∗ i − 1 , v i } . The i th iteration computes v ∗ i using project orthogonal ( v i , [ v ∗ 1 , . . . , v ∗ i − 1 ]). There are scalars α i 1 , α i 2 , . . . , α i , i − 1 such that v i = α 1 i v ∗ 1 + · · · + α i − 1 , i v ∗ i − 1 + v ∗ i This equation shows that any linear combination of v ∗ 1 , v ∗ 2 . . . , v ∗ i − 1 , v i can be transformed into a linear combination of v ∗ 1 , v ∗ 2 . . . , v ∗ i − 1 , v ∗ i and vice versa. QED

Order in orthogonalize Order matters! Suppose you run the procedure orthogonalize twice, once with a list of vectors and once with the reverse of that list. The output lists will not be the reverses of each other. Contrast with project orthogonal(b, vlist) . The projection of a vector b orthogonal to a vector space is unique, so in principle the order of vectors in vlist doesn’t a ff ect the output of project orthogonal(b, vlist) .

Matrix form for orthogonalize 2 3 α 0 2 3 2 3 . For project orthogonal , we had . 6 7 . 4 b 5 = 4 v 0 b ⊥ · · · v n 6 7 5 6 7 For orthogonalize , we have α n 4 5 1 2 3 2 3 4 v 0 4 v ∗ 5 = 5 ⇥ ⇤ 1 2 3 1 α 01 α 02 α 03 0 2 3 2 3 1 4 v 0 4 v ∗ α 12 α 13 5 = v ∗ v ∗ v ∗ 6 7 v 1 v 2 v 3 6 7 2 3 2 3 0 1 2 3 5 1  α 01 α 23 4 5 � 4 v 1 5 = 4 v ∗ v ∗ 1 0 1 5 1 2 3 2 3 2 3 α 02 4 v 2 4 v ∗ 5 = v ∗ 2 v ∗ α 12 0 1 2 5 4 5 1 2 3 α 03 2 3 2 3 α 13 4 v 3 5 = 4 v ∗ v ∗ v ∗ v ∗ 6 7 0 1 2 3 5 6 7 α 23 4 5 1

Example of matrix form for orthogonalize Example: for vlist consisting of vectors 2 2 3 2 1 3 2 1 3 v 0 = 5 , v 1 = 5 , v 2 = 0 2 0 4 4 4 5 0 2 2 we saw that the output list vstarlist of orthogonal vectors consists of 2 3 2 3 2 3 2 0 0 v ∗ 5 , v ∗ 5 , v ∗ 0 = 0 1 = 2 2 = � 1 4 4 4 5 0 2 1 The corresponding matrix equation is 2 3 2 2 0 0 3 2 1 0 . 5 0 . 5 3 4 v 0 5 = v 1 v 2 0 2 � 1 1 0 . 5 4 5 4 5 0 2 1 1