Least Squares and Data Fitting Data fitting How do we best fit a - - PowerPoint PPT Presentation

Least Squares and Data Fitting

How do we best fit a set of data points?

Data fitting

Given 𝑛 data points { 𝑢!, 𝑧! , … , 𝑢", 𝑧" }, we want to find the function

𝑧 = 𝑦! + 𝑦" 𝑢

that best fit the data (or better, we want to find the coefficients 𝑦#, 𝑦!). Thinking geometrically, we can think “what is the line that most nearly passes through all the points?”

Linear Least Squares

1) Fitting with a line

Given 𝑛 data points { 𝑢!, 𝑧! , … , 𝑢", 𝑧" }, we want to find 𝑦# and 𝑦! such that 𝑧$ = 𝑦# + 𝑦! 𝑢$ ∀𝑗 ∈ 1, 𝑛

• r in matrix form:

Note that this system of linear equations has more equations than unknowns – OVERDETERMINED SYSTEMS 𝒏×𝒐 𝒐×𝟐 𝒏×𝟐 1 𝑢! ⋮ ⋮ 1 𝑢" 𝑦# 𝑦! = 𝑧! ⋮ 𝑧"

𝑩 𝒚 = 𝒄

We want to find the appropriate linear combination of the columns of 𝑩 that makes up the vector 𝒄. If a solution exists that satisfies 𝑩 𝒚 = 𝒄 then 𝒄 ∈ 𝑠𝑏𝑜𝑕𝑓(𝑩)

Linear Least Squares

• In most cases, 𝒄 ∉ 𝑠𝑏𝑜𝑕𝑓(𝑩) and 𝑩 𝒚 = 𝒄 does not have an

exact solution!

• Therefore, an overdetermined system is better expressed as

𝑩 𝒚 ≅ 𝒄

Linear Least Squares

• Least Squares: find the solution 𝒚 that minimizes the residual

𝒔 = 𝒄 − 𝑩 𝒚

• Let’s define the function 𝜚 as the square of the 2-norm of the residual

𝜚 𝒚 = 𝒄 − 𝑩 𝒚 %

%

Linear Least Squares

• Least Squares: find the solution 𝒚 that minimizes the residual

𝒔 = 𝒄 − 𝑩 𝒚

• Let’s define the function 𝜚 as the square of the 2-norm of the residual

𝜚 𝒚 = 𝒄 − 𝑩 𝒚 %

%

• Then the least squares problem becomes

min

𝒚 𝜚 (𝒚)

• Suppose 𝜚: ℛ" → ℛ is a smooth function, then 𝜚 𝒚 reaches a (local)

maximum or minimum at a point 𝒚∗ ∈ ℛ" only if ∇𝜚 𝒚∗ = 0

How to find the minimizer?

• To minimize the 2-norm of the residual vector

min

𝒚 𝜚 𝒚 =

𝒄 − 𝑩 𝒚 %

%

𝜚 𝒚 = (𝒄 − 𝑩 𝒚)((𝒄 − 𝑩 𝒚) ∇𝜚 𝒚 = 2(𝑩( 𝒄 − 𝑩(𝑩 𝒚) First order necessary condition: ∇𝜚 𝒚 = 0 → 𝑩( 𝒄 − 𝑩(𝑩 𝒚 = 𝟏 → 𝑩(𝑩 𝒚 = 𝑩( 𝒄 Second order sufficient condition: 𝐸%𝜚 𝒚 = 2𝑩(𝑩 2𝑩(𝑩 is a positive semi-definite matrix → the solution is a minimum Normal Equations – solve a linear system of equations

Linear Least Squares (another approach)

• Find 𝒛 = 𝑩 𝒚 which is closest to the vector 𝒄
• What is the vector 𝒛 = 𝑩 𝒚 ∈ 𝑠𝑏𝑜𝑕𝑓(𝑩) that is closest to vector 𝒛 in

the Euclidean norm?

When 𝒔 = 𝒄 − 𝒛 = 𝒄 − 𝑩 𝒚 is orthogonal to all columns of 𝑩, then 𝒛 is closest to 𝒄 𝑩𝑼𝒔 = 𝑩𝑼(𝒄 − 𝑩 𝒚)=0 𝑩"𝑩 𝒚 = 𝑩" 𝒄

Summary:

• 𝑩 is a 𝑛×𝑜 matrix, where 𝑛 > 𝑜.
• 𝑛 is the number of data pair points. 𝑜 is the number of parameters of the

“best fit” function.

• Linear Least Squares problem 𝑩 𝒚 ≅ 𝒄 always has solution.
• The Linear Least Squares solution 𝒚 minimizes the square of the 2-norm
• f the residual:

min

𝒚

𝒄 − 𝑩 𝒚 %

%

• One method to solve the minimization problem is to solve the system of

Normal Equations 𝑩(𝑩 𝒚 = 𝑩( 𝒄

• Let’s see some examples and discuss the limitations of this method.

Example:

Solve: 𝑩#𝑩 𝒚 = 𝑩# 𝒄

• Does not need to be a line! For example, here we are fitting the data

using a quadratic curve.

Data fitting - not always a line fit!

Linear Least Squares: The problem is linear in its coefficients!

Another examples

We want to find the coefficients of the quadratic function that best fits the data points:

We would not want our “fit” curve to pass through the data points exactly as we are looking to model the general trend and not capture the noise.

𝑧 = 𝑦! + 𝑦" 𝑢 + 𝑦# 𝑢#

Data fitting

1 𝑢! 𝑢!

"

⋮ ⋮ ⋮ 1 𝑢# 𝑢#

"

𝑦$ 𝑦! 𝑦" = 𝑧! ⋮ 𝑧#

(𝑢$,𝑧$) Solve: 𝑩#𝑩 𝒚 = 𝑩# 𝒄

Which function is not suitable for linear least squares? A) 𝑧 = 𝑏 + 𝑐 𝑦 + 𝑑 𝑦# + 𝑒 𝑦% B) 𝑧 = 𝑦 𝑏 + 𝑐 𝑦 + 𝑑 𝑦# + 𝑒 𝑦% C) 𝑧 = 𝑏 sin 𝑦 + 𝑐/ cos 𝑦 D) 𝑧 = 𝑏 sin 𝑦 + 𝑦/ cos 𝑐𝑦 E) 𝑧 = 𝑏 𝑓&#' + 𝑐 𝑓#'

Computational Cost

𝑩#𝑩 𝒚 = 𝑩# 𝒄

• Compute 𝑩(𝑩: 𝑃 𝑛𝑜%
• Factorize 𝑩(𝑩: LU → 𝑃

% 2 𝑜2 , Cholesky → 𝑃 ! 2 𝑜2

• Solve 𝑃 𝑜%
• Since 𝑛 > 𝑜 the overall cost is 𝑃 𝑛𝑜%

Short questions

Given the data in the table below, which of the plots shows the line of best fit in terms of least squares? A) B) C) D)

Short questions

Given the data in the table below, and the least squares model 𝑧 = 𝑑! + 𝑑% sin 𝑢𝜌 + 𝑑2 sin 𝑢𝜌/2 + 𝑑3 sin 𝑢𝜌/4 written in matrix form as determine the entry 𝐵%2 of the matrix 𝑩. Note that indices start with 1. A) −1.0 B) 1.0 C) − 0.7 D) 0.7 E) 0.0

Solving Linear Least Squares with SVD

𝑩 is a 𝑛×𝑜 matrix where 𝑛 > 𝑜 (more points to fit than coefficient to be determined) Normal Equations: 𝑩!𝑩 𝒚 = 𝑩! 𝒄

• The solution 𝑩 𝒚 ≅ 𝒄 is unique if and only if 𝑠𝑏𝑜𝑙 𝐁 = 𝑜

(𝑩 is full column rank)

• 𝑠𝑏𝑜𝑙 𝐁 = 𝑜 → columns of 𝑩 are linearly independent → 𝑜 non-zero

singular values → 𝑩! 𝑩 has only positive eigenvalues → 𝑩!𝑩 is a symmetric and positive definite matrix → 𝑩!𝑩 is invertible 𝒚 = 𝑩!𝑩 "𝟐𝑩! 𝒄

• If 𝑠𝑏𝑜𝑙 𝐁 < 𝑜, then 𝑩 is rank-deficient, and solution of linear least squares

problem is not unique.

What we have learned so far…

Condition number for Normal Equations

Finding the least square solution of 𝑩 𝒚 ≅ 𝒄 (where 𝑩 is full rank matrix) using the Normal Equations 𝑩(𝑩 𝒚 = 𝑩( 𝒄 has some advantages, since we are solving a square system of linear equations with a symmetric matrix (and hence it is possible to use decompositions such as Cholesky Factorization) However, the normal equations tend to worsen the conditioning of the matrix. 𝑑𝑝𝑜𝑒 𝑩(𝑩 = (𝑑𝑝𝑜𝑒 𝑩 )% How can we solve the least square problem without squaring the condition of the matrix?

SVD to solve linear least squares problems

We want to find the least square solution of 𝑩 𝒚 ≅ 𝒄, where 𝑩 = 𝑽 𝚻 𝑾𝑼

• r better expressed in reduced form: 𝑩 = 𝑽5 𝚻𝑺 𝑾𝑼

𝑩 = ⋮ … ⋮ 𝒗# … 𝒗$ ⋮ … ⋮ 𝜏# ⋱ 𝜏% ⋮ … 𝐰#

"

… ⋮ ⋮ ⋮ … 𝐰%

"

…

𝑩 is a 𝑛×𝑜 rectangular matrix where 𝑛 > 𝑜, and hence the SVD decomposition is given by:

Recall Reduced SVD

𝑩 = 𝑽4 𝚻𝑺 𝑾𝑼

𝑛×𝑜 𝑛×𝑜 𝑜×𝑜 𝑜×𝑜

𝑛 > 𝑜

SVD to solve linear least squares problems

We want to find the least square solution of 𝑩 𝒚 ≅ 𝒄, where 𝑩 = 𝑽$ 𝚻𝑺 𝑾𝑼 Normal equations: 𝑩"𝑩 𝒚 = 𝑩" 𝒄 ⟶ 𝑽& 𝚻𝑺 𝑾" " 𝑽& 𝚻𝑺 𝑾" 𝒚 = 𝑽& 𝚻𝑺 𝑾" "𝒄

𝑩 = ⋮ … ⋮ 𝒗! … 𝒗A ⋮ … ⋮ 𝜏! ⋱ 𝜏A … 𝐰!

(

… ⋮ ⋮ ⋮ … 𝐰A

(

…

𝑩 = 𝑽4 𝚻𝑺 𝑾𝑼

𝑾 𝚻𝑺𝑽&

" 𝑽& 𝚻𝑺 𝑾" 𝒚 = 𝑾 𝚻𝑺𝑽& "𝒄

𝑾 𝚻𝑺 𝚻𝑺 𝑾"𝒚 = 𝑾 𝚻𝑺𝑽&

"𝒄

𝚻𝑺

( 𝑾"𝒚 = 𝚻𝑺 𝑽& "𝒄

When can we take the inverse of the singular matrix?

𝚻𝑺

" 𝑾#𝒚 = 𝚻𝑺 𝑽$ #𝒄

1) Full rank matrix (𝜏$ ≠ 0 ∀𝑗):

𝑾#𝒚 = 𝚻𝑺

%&𝑽$ #𝒄

𝒚 = 𝑾 𝚻𝑺

"'𝑽$ ! 𝒄

Unique solution:

𝑜×𝑜 𝑜×𝑜 𝑜×𝑛 𝑛×1 𝑜×1

rank 𝑩 = 𝑜

2) Rank deficient matrix ( rank 𝑩 = 𝑠 < 𝑜 )

Solution is not unique!!

Find solution 𝒚 such that min

𝒚 𝜚 𝒚 =

𝒄 − 𝑩 𝒚 "

"

and also 𝚻𝑺

" 𝑾#𝒚 = 𝚻𝑺 𝑽$ #𝒄

min

𝒚

𝒚 𝟑

2) Rank deficient matrix (continue)

Change of variables: Set 𝑾#𝒚 = 𝒛 and then solve 𝚻𝑺 𝒛 = 𝑽$

#𝒄 for the variable 𝒛

𝜏# ⋱ 𝜏) ⋱ 𝑧# ⋮ 𝑧) 𝑧)*# ⋮ 𝑧% = 𝒗#

"𝒄

⋮ 𝒗)

"𝒄

𝒗)*#

"

𝒄 ⋮ 𝒗%

"𝒄

𝑧+ = 𝒗+

"𝒄

𝜏+ 𝑗 = 1,2, … , 𝑠 What do we do when 𝑗 > 𝑠? Which choice of 𝑧+ will minimize 𝒚 𝟑 = 𝑾 𝒛 𝟑? 𝑧+ = 0, 𝑗 = 𝑠 + 1, … , 𝑜 Set

We want to find the solution 𝒚 that satisfies 𝚻𝑺

" 𝑾#𝒚 = 𝚻𝑺 𝑽$ #𝒄 and also satisfies

min

𝒚

𝒚 𝟑 Evaluate 𝒚 = 𝑾𝒛 = ⋮ … ⋮ 𝒘& … 𝒘) ⋮ … ⋮ 𝑧& 𝑧" ⋮ 𝑧) 𝒚 = 9

*+& )

𝑧* 𝒘𝒋 = 9

*+&

• !./

) (𝒗* #𝒄)

𝜏* 𝒘𝒋

Solving Least Squares Problem with SVD (summary)

• Find 𝒚 that satisfies min

𝒚

𝒄 − 𝑩 𝒚 )

)

• Find 𝒛 that satisfies min

𝒛

𝚻𝑺 𝒛 − 𝑽$

#𝒄 " "

• Propose 𝒛 that is solution of 𝚻𝑺 𝒛 = 𝑽5

(𝒄

• Evaluate: 𝒜 = 𝑽5

(𝒄

• Set: 𝑧$ = h

O" P" , if 𝜏$ ≠ 0

0, otherwise 𝑗 = 1, … , 𝑜

• Then compute 𝒚 = 𝑾 𝒛

Cost:

𝑛 𝑜 𝑜 𝑜# Cost of SVD: 𝑃(𝑛 𝑜#)

• If 𝜏$≠ 0 for ∀𝑗 = 1, … , 𝑜, then the solution 𝒛 = 𝑾

𝚻𝑺

Q!𝑽5 ( 𝒄 is

unique (and not a “choice”).

• If at least one of the singular values is zero, then the proposed solution 𝒛 is

the one with the smallest 2-norm ( 𝒛 % is minimal ) that minimizes the 2-norm of the residual 𝚻𝑺 𝒛 − 𝑽5

(𝒄 %

• Since 𝒚 % =

𝑾 𝒛 %= 𝒛 %, then the solution 𝒚 is also the one with the smallest 2-norm ( 𝒚 % is minimal ) for all possible 𝒚 for which 𝑩𝒚 − 𝒄 % is minimal.

Solving Least Squares Problem with SVD (summary)

Solve 𝑩 𝒚 ≅ 𝒄 or 𝑽> 𝚻𝑺𝑾𝑼𝒚 ≅ 𝒄 𝒚 ≅ 𝑾 𝚻𝑺 A 𝑽>

B 𝒄

Solving Least Squares Problem with SVD (summary)

Consider solving the least squares problem 𝑩 𝒚 ≅ 𝒄, where the singular value decomposition of the matrix 𝑩 = 𝑽 𝚻 𝑾𝑼𝒚 is: Determine 𝒄 − 𝑩 𝒚 (

Example:

Example

Suppose you have 𝑩 = 𝑽 𝚻 𝑾𝑼𝒚 calculated. What is the cost of solving min

𝒚

𝒄 − 𝑩 𝒚 %

% ?

A) 𝑃(𝑜) B) 𝑃( 𝑜%) C) 𝑃(𝑛𝑜) D) 𝑃 𝑛 E) 𝑃( 𝑛%)