ON-Bases and Least Square Method Artem Los (arteml@kth.se) - - PowerPoint PPT Presentation

▶
on bases and least square method
SMART_READER_LITE
LIVE PREVIEW

ON-Bases and Least Square Method Artem Los (arteml@kth.se) - - PowerPoint PPT Presentation

Nov 01, 2022 279 likes •659 views

ON-Bases and Least Square Method Artem Los (arteml@kth.se) February 21th, 2017 Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 1 / 17 Overview Projections onto Subspaces 1 Orthogonal Complement 2

slide-1
SLIDE 1

ON-Bases and Least Square Method

Artem Los (arteml@kth.se) February 21th, 2017

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 1 / 17

slide-2
SLIDE 2

Overview

1

Projections onto Subspaces

2

Orthogonal Complement

3

Gram-Schmidt Method

4

Least Square Regression

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 2 / 17

slide-3
SLIDE 3

Projections onto Subspaces

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 3 / 17

slide-4
SLIDE 4

Projecting a vector onto subspace

Projection of vector b onto a plane π is the best approximation of b in π, defined as: projπ =

  • b ·

v1 || v1||2 v1 +

  • b ·

v2 || v2||2 v2

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 4 / 17

slide-5
SLIDE 5

Projecting a vector onto subspace

Projection of vector b onto a plane π is the best approximation of b in π, defined as: projπ =

  • b ·

v1 || v1||2 v1 +

  • b ·

v2 || v2||2 v2 We’ve assumed that π = span{ v1, v2}. The same pattern is applied to hyper planes, etc.

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 4 / 17

slide-6
SLIDE 6

Example

Problem. Find the projection of x = (2, 3, 5, 6) onto π = s(1, −1, −1, 1) + t(1, 2, 1, 2) : s, t ∈ R

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 5 / 17

slide-7
SLIDE 7

Example

Problem. Find the projection of x = (2, 3, 5, 6) onto π = s(1, −1, −1, 1) + t(1, 2, 1, 2) : s, t ∈ R Step 1: Find projπ( x) using the definition projπ =

  • b ·

v1 || v1||2 v1 +

  • b ·

v2 || v2||2 v2 =

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 5 / 17

slide-8
SLIDE 8

Example

Problem. Find the projection of x = (2, 3, 5, 6) onto π = s(1, −1, −1, 1) + t(1, 2, 1, 2) : s, t ∈ R Step 1: Find projπ( x) using the definition projπ =

  • b ·

v1 || v1||2 v1 +

  • b ·

v2 || v2||2 v2 = projπ x = (2, 3, 5, 6) · (1, −1, −1, 1) ||(1, −1, −1, 1)||2 (1, −1, −1, 1)+ +(2, 3, 5, 6) · (1, 2, 1, 2) ||(1, 2, 1, 2)||2 (1, 2, 1, 2) =

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 5 / 17

slide-9
SLIDE 9

Example

Problem. Find the projection of x = (2, 3, 5, 6) onto π = s(1, −1, −1, 1) + t(1, 2, 1, 2) : s, t ∈ R Step 1: Find projπ( x) using the definition projπ =

  • b ·

v1 || v1||2 v1 +

  • b ·

v2 || v2||2 v2 = projπ x = (2, 3, 5, 6) · (1, −1, −1, 1) ||(1, −1, −1, 1)||2 (1, −1, −1, 1)+ +(2, 3, 5, 6) · (1, 2, 1, 2) ||(1, 2, 1, 2)||2 (1, 2, 1, 2) = = 5 2, 5, 5 2, 5

  • Artem Los (arteml@kth.se)

ON-Bases and Least Square Method February 21th, 2017 5 / 17

slide-10
SLIDE 10

Orthogonal Complement

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 6 / 17

slide-11
SLIDE 11

Orthogonal complement

Relationship with row space

For all matrices A, the null space of A is an orthogonal complement to the row space of A.

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 7 / 17

slide-12
SLIDE 12

Orthogonal complement

Relationship with row space

For all matrices A, the null space of A is an orthogonal complement to the row space of A. How about AT?

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 7 / 17

slide-13
SLIDE 13

Example

Problem. Find the basis of the orthogonal complement of S = span{(1, 2, −1)}, i.e. S⊥.

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 8 / 17

slide-14
SLIDE 14

Example

Problem. Find the basis of the orthogonal complement of S = span{(1, 2, −1)}, i.e. S⊥. Step 1: Transform into a plane.

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 8 / 17

slide-15
SLIDE 15

Example

Problem. Find the basis of the orthogonal complement of S = span{(1, 2, −1)}, i.e. S⊥. Step 1: Transform into a plane. S = span{(1, 2, −1)} means the equation of the plane is x1 + 2x2 − x3.

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 8 / 17

slide-16
SLIDE 16

Example

Problem. Find the basis of the orthogonal complement of S = span{(1, 2, −1)}, i.e. S⊥. Step 1: Transform into a plane. S = span{(1, 2, −1)} means the equation of the plane is x1 + 2x2 − x3. Step 2: Insert this into a matrix (interpret as row space) Since we want to find the null space, we solve x1 + 2x2 − x3 = 0.

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 8 / 17

slide-17
SLIDE 17

Example

Problem. Find the basis of the orthogonal complement of S = span{(1, 2, −1)}, i.e. S⊥. Step 1: Transform into a plane. S = span{(1, 2, −1)} means the equation of the plane is x1 + 2x2 − x3. Step 2: Insert this into a matrix (interpret as row space) Since we want to find the null space, we solve x1 + 2x2 − x3 = 0. Step 3: Use parametrisation to get the solution space (here it’s null space) s(−2, 1, 0) + t(1, 0, 1) s, t ∈ R

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 8 / 17

slide-18
SLIDE 18

Gram-Schmidt Method

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 9 / 17

slide-19
SLIDE 19

Find orthonormal basis using Gram-Schmidt Method

Goal. We want to find an orthonormal basis given { v1, v2, . . . , vk}. We will see later that this is useful.

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 10 / 17

slide-20
SLIDE 20

Find orthonormal basis using Gram-Schmidt Method

Goal. We want to find an orthonormal basis given { v1, v2, . . . , vk}. We will see later that this is useful. Algorithm. Step1 : u1 = v1 Step2 : u2 = v2 −

v2· u1 || u1||2

u1 Step2 : u3 = v3 −

v3· u1 || u1||2

u1 −

v3· u2 || u2||2

u2 . . . r times

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 10 / 17

slide-21
SLIDE 21

Example

Problem. We are given a basis spanned by (1, 1, 1), (1, 1, 0), (1, 0, 0). Find an orthonormal basis.

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 11 / 17

slide-22
SLIDE 22

Example

Problem. We are given a basis spanned by (1, 1, 1), (1, 1, 0), (1, 0, 0). Find an orthonormal basis. Step 1: Find an orthogonal basis using Gram-Schmidt process.

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 11 / 17

slide-23
SLIDE 23

Example

Problem. We are given a basis spanned by (1, 1, 1), (1, 1, 0), (1, 0, 0). Find an orthonormal basis. Step 1: Find an orthogonal basis using Gram-Schmidt process. Step 2: Normalize the new vectors to get an orthonormal basis.

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 11 / 17

slide-24
SLIDE 24

Example

Problem. We are given a basis spanned by (1, 1, 1), (1, 1, 0), (1, 0, 0). Find an orthonormal basis. Step 1: Find an orthogonal basis using Gram-Schmidt process. Step 2: Normalize the new vectors to get an orthonormal basis. Example in Python

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 11 / 17

slide-25
SLIDE 25

Least Square Regression

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 12 / 17

slide-26
SLIDE 26

Definition

Least Square Method

The method of least squares is a standard approach in regression analysis to the approximate solution of overdetermined systems, i.e., sets of equations in which there are more equations than unknowns.

(From https://en.wikipedia.org/wiki/Least_squares) Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 13 / 17

slide-27
SLIDE 27

Definition

Least Square Method

The method of least squares is a standard approach in regression analysis to the approximate solution of overdetermined systems, i.e., sets of equations in which there are more equations than unknowns.

(From https://en.wikipedia.org/wiki/Least_squares)

Examples of usage: Find the equation of straight line going through a set of points (eg. from an experiment). Fitting a curve to set of points

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 13 / 17

slide-28
SLIDE 28

The trick of solving overdetermined systems

Let A be the matrix with more equations than unknowns (i.e.

  • verdetermined). Then, to minimize ||A

x − b|| is the same as solving ATA x = AT

  • b. So,

ATA x = AT b = ⇒

  • x = (ATA)−1AT

b

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 14 / 17

slide-29
SLIDE 29

Example

Problem. We want to fit y = a + bt2 and we are given five data points: (−2, 1), (−1, 1), (0, 2), (1, 3), (2, −2)

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 15 / 17

slide-30
SLIDE 30

Example

Problem. We want to fit y = a + bt2 and we are given five data points: (−2, 1), (−1, 1), (0, 2), (1, 3), (2, −2) Step 1: Find A       1 4 1 1 1 1 1 1 4      

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 15 / 17

slide-31
SLIDE 31

Example

Problem. We want to fit y = a + bt2 and we are given five data points: (−2, 1), (−1, 1), (0, 2), (1, 3), (2, −2) Step 1: Find A       1 4 1 1 1 1 1 1 4       Step 2: Find ATA 1 1 1 1 1 4 1 1 4

     1 4 1 1 1 1 1 1 4       = 5 10 10 34

  • Artem Los (arteml@kth.se)

ON-Bases and Least Square Method February 21th, 2017 15 / 17

slide-32
SLIDE 32

Example

Problem. We want to fit y = a + bt2 and we are given five data points: (−2, 1), (−1, 1), (0, 2), (1, 3), (2, −2)

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 16 / 17

slide-33
SLIDE 33

Example

Problem. We want to fit y = a + bt2 and we are given five data points: (−2, 1), (−1, 1), (0, 2), (1, 3), (2, −2) Step 3: Find AT b 1 1 1 1 1 4 1 1 4

     1 1 2 3 −2       = 5

  • Artem Los (arteml@kth.se)

ON-Bases and Least Square Method February 21th, 2017 16 / 17

slide-34
SLIDE 34

Example

Problem. We want to fit y = a + bt2 and we are given five data points: (−2, 1), (−1, 1), (0, 2), (1, 3), (2, −2) Step 3: Find AT b 1 1 1 1 1 4 1 1 4

     1 1 2 3 −2       = 5

  • Step 4: Solve ATA = AT

b 5 10 10 34

  • x =

5

  • Artem Los (arteml@kth.se)

ON-Bases and Least Square Method February 21th, 2017 16 / 17

slide-35
SLIDE 35

Example

Problem. We want to fit y = a + bt2 and we are given five data points: (−2, 1), (−1, 1), (0, 2), (1, 3), (2, −2) Step 3: Find AT b 1 1 1 1 1 4 1 1 4

     1 1 2 3 −2       = 5

  • Step 4: Solve ATA = AT

b 5 10 10 34

  • x =

5

  • Example in Python

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 16 / 17

slide-36
SLIDE 36

Application for Least Squared

  • Question. Find a relationship between the temperature of Artem’s tea

cup and time. How long time is the tea still drinkable?

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 17 / 17

slide-37
SLIDE 37

Application for Least Squared

  • Question. Find a relationship between the temperature of Artem’s tea

cup and time. How long time is the tea still drinkable? Data can be found here

Artem Los (arteml@kth.se) ON-Bases and Least Square Method February 21th, 2017 17 / 17