ON-Bases and Least Square Method Artem Los (arteml@kth.se) February 21th, 2017 Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 1 / 17
Overview Projections onto Subspaces 1 Orthogonal Complement 2 Gram-Schmidt Method 3 Least Square Regression 4 Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 2 / 17
Projections onto Subspaces Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 3 / 17
Projecting a vector onto subspace Projection of vector � b onto a plane π is the best approximation of b in π , defined as: � � b · � v 1 b · � v 2 proj π = v 1 || 2 � v 1 + v 2 || 2 � v 2 || � || � Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 4 / 17
Projecting a vector onto subspace Projection of vector � b onto a plane π is the best approximation of b in π , defined as: � � b · � v 1 b · � v 2 proj π = v 1 || 2 � v 1 + v 2 || 2 � v 2 || � || � We’ve assumed that π = span { � v 1 , � v 2 } . The same pattern is applied to hyper planes, etc. Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 4 / 17
Example Problem. Find the projection of � x = (2 , 3 , 5 , 6) onto π = s (1 , − 1 , − 1 , 1) + t (1 , 2 , 1 , 2) : s , t ∈ R Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 5 / 17
Example Problem. Find the projection of � x = (2 , 3 , 5 , 6) onto π = s (1 , − 1 , − 1 , 1) + t (1 , 2 , 1 , 2) : s , t ∈ R Step 1: Find proj π ( � x ) using the definition � � b · � b · � v 1 v 2 proj π = v 1 || 2 � v 1 + v 2 || 2 � v 2 = || � || � Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 5 / 17
Example Problem. Find the projection of � x = (2 , 3 , 5 , 6) onto π = s (1 , − 1 , − 1 , 1) + t (1 , 2 , 1 , 2) : s , t ∈ R Step 1: Find proj π ( � x ) using the definition � � b · � b · � v 1 v 2 proj π = v 1 || 2 � v 1 + v 2 || 2 � v 2 = || � || � x = (2 , 3 , 5 , 6) · (1 , − 1 , − 1 , 1) proj π � (1 , − 1 , − 1 , 1)+ || (1 , − 1 , − 1 , 1) || 2 +(2 , 3 , 5 , 6) · (1 , 2 , 1 , 2) (1 , 2 , 1 , 2) = || (1 , 2 , 1 , 2) || 2 Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 5 / 17
Example Problem. Find the projection of � x = (2 , 3 , 5 , 6) onto π = s (1 , − 1 , − 1 , 1) + t (1 , 2 , 1 , 2) : s , t ∈ R Step 1: Find proj π ( � x ) using the definition � � b · � b · � v 1 v 2 proj π = v 1 || 2 � v 1 + v 2 || 2 � v 2 = || � || � x = (2 , 3 , 5 , 6) · (1 , − 1 , − 1 , 1) proj π � (1 , − 1 , − 1 , 1)+ || (1 , − 1 , − 1 , 1) || 2 +(2 , 3 , 5 , 6) · (1 , 2 , 1 , 2) (1 , 2 , 1 , 2) = || (1 , 2 , 1 , 2) || 2 � 5 2 , 5 , 5 � = 2 , 5 Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 5 / 17
Orthogonal Complement Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 6 / 17
Orthogonal complement Relationship with row space For all matrices A , the null space of A is an orthogonal complement to the row space of A . Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 7 / 17
Orthogonal complement Relationship with row space For all matrices A , the null space of A is an orthogonal complement to the row space of A . How about A T ? Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 7 / 17
Example Find the basis of the orthogonal complement of Problem. S = span { (1 , 2 , − 1) } , i.e. S ⊥ . Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 8 / 17
Example Find the basis of the orthogonal complement of Problem. S = span { (1 , 2 , − 1) } , i.e. S ⊥ . Step 1: Transform into a plane. Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 8 / 17
Example Find the basis of the orthogonal complement of Problem. S = span { (1 , 2 , − 1) } , i.e. S ⊥ . Step 1: Transform into a plane. S = span { (1 , 2 , − 1) } means the equation of the plane is x 1 + 2 x 2 − x 3 . Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 8 / 17
Example Find the basis of the orthogonal complement of Problem. S = span { (1 , 2 , − 1) } , i.e. S ⊥ . Step 1: Transform into a plane. S = span { (1 , 2 , − 1) } means the equation of the plane is x 1 + 2 x 2 − x 3 . Step 2: Insert this into a matrix (interpret as row space) Since we want to find the null space, we solve x 1 + 2 x 2 − x 3 = 0. Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 8 / 17
Example Find the basis of the orthogonal complement of Problem. S = span { (1 , 2 , − 1) } , i.e. S ⊥ . Step 1: Transform into a plane. S = span { (1 , 2 , − 1) } means the equation of the plane is x 1 + 2 x 2 − x 3 . Step 2: Insert this into a matrix (interpret as row space) Since we want to find the null space, we solve x 1 + 2 x 2 − x 3 = 0. Step 3: Use parametrisation to get the solution space (here it’s null space) s ( − 2 , 1 , 0) + t (1 , 0 , 1) s , t ∈ R Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 8 / 17
Gram-Schmidt Method Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 9 / 17
Find orthonormal basis using Gram-Schmidt Method Goal. We want to find an orthonormal basis given { � v 1 , � v 2 , . . . , � v k } . We will see later that this is useful. Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 10 / 17
Find orthonormal basis using Gram-Schmidt Method Goal. We want to find an orthonormal basis given { � v 1 , � v 2 , . . . , � v k } . We will see later that this is useful. Algorithm. Step1 : � u 1 = � v 1 v 2 − � v 2 · � u 1 Step2 : � u 2 = � u 1 || 2 � u 1 || � v 3 − � v 3 · � u 1 − � v 3 · � u 1 u 2 Step2 : � u 3 = � u 1 || 2 � u 2 || 2 � u 2 || � || � . . . r times Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 10 / 17
Example Problem. We are given a basis spanned by (1 , 1 , 1) , (1 , 1 , 0) , (1 , 0 , 0). Find an orthonormal basis. Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 11 / 17
Example Problem. We are given a basis spanned by (1 , 1 , 1) , (1 , 1 , 0) , (1 , 0 , 0). Find an orthonormal basis. Step 1: Find an orthogonal basis using Gram-Schmidt process. Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 11 / 17
Example Problem. We are given a basis spanned by (1 , 1 , 1) , (1 , 1 , 0) , (1 , 0 , 0). Find an orthonormal basis. Step 1: Find an orthogonal basis using Gram-Schmidt process. Step 2: Normalize the new vectors to get an orthonormal basis. Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 11 / 17
Example Problem. We are given a basis spanned by (1 , 1 , 1) , (1 , 1 , 0) , (1 , 0 , 0). Find an orthonormal basis. Step 1: Find an orthogonal basis using Gram-Schmidt process. Step 2: Normalize the new vectors to get an orthonormal basis. Example in Python Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 11 / 17
Least Square Regression Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 12 / 17
Definition Least Square Method The method of least squares is a standard approach in regression analysis to the approximate solution of overdetermined systems, i.e., sets of equations in which there are more equations than unknowns . (From https://en.wikipedia.org/wiki/Least_squares ) Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 13 / 17
Definition Least Square Method The method of least squares is a standard approach in regression analysis to the approximate solution of overdetermined systems, i.e., sets of equations in which there are more equations than unknowns . (From https://en.wikipedia.org/wiki/Least_squares ) Examples of usage: Find the equation of straight line going through a set of points (eg. from an experiment). Fitting a curve to set of points Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 13 / 17
The trick of solving overdetermined systems Let A be the matrix with more equations than unknowns (i.e. x − � overdetermined). Then, to minimize || A � b || is the same as solving x = A T � A T A � b . So, x = A T � x = ( A T A ) − 1 A T � A T A � b = ⇒ � b Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 14 / 17
Example We want to fit y = a + bt 2 and we are given five data points: Problem. ( − 2 , 1) , ( − 1 , 1) , (0 , 2) , (1 , 3) , (2 , − 2) Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 15 / 17
Example We want to fit y = a + bt 2 and we are given five data points: Problem. ( − 2 , 1) , ( − 1 , 1) , (0 , 2) , (1 , 3) , (2 , − 2) Step 1: Find A 1 4 1 1 1 0 1 1 1 4 Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 15 / 17
Recommend
More recommend
