CS475 / CS675 Lecture 10: June 2, 2016 Least Squares Problems - - PowerPoint PPT Presentation

▶

Apr 02, 2023 167 likes •379 views

CS475 / CS675 Lecture 10: June 2, 2016 Least Squares Problems Reading: [TB] Chapt 11 CS475/CS675 (c) 2016 P. Poupart & J. Wan 1 Least Squares Problems First posed and formulated by Gauss. Surveyors tried to identify boundaries by

SLIDE 1

CS475 / CS675 Lecture 10: June 2, 2016

Least Squares Problems Reading: [TB] Chapt 11

SLIDE 2

Least Squares Problems

First posed and formulated by Gauss.
Surveyors tried to identify boundaries by measuring

certain angles and distances from known landmarks.

To update the location of landmarks, new

measurements of angles and distances between landmarks are made.

SLIDE 3

Surveying Example

Given a set of old locations
, find correction
such that
better

match new measurements

Picture:

SLIDE 4

Surveying Example

Non‐linear constraint:

cos

Suppose
– Multiply through the denominator

– Multiply out all the terms to get a quartic polynomial in all ‐variables – Throw away all terms containing , , ⟹ linear constraint

SLIDE 5

Surveying example

Collect all linear constraints for all angles and

distance measurements

⟹ overdetermined linear system

In general:

constraints ⋯ ⋮ ⋯

bservations

Picture:

SLIDE 6

Overdetermined system

In general:
Idea: minimize the residual vector

– Optimization problem:

Least squares (LS) problems

SLIDE 7

Solving LS Problems

Geometric interpretation:

SLIDE 8

Solving LS Problems

Theorem: Let

, ,

. A vector

minimizes

if and only if
CS475/CS675 (c) 2016 P. Poupart & J. Wan

SLIDE 9

Pseudoinverse

Def:
is called

the pseudoinverse of

The least squares solution is given by
Why is the minimizer of

SLIDE 10

Pseudoinverse

Let
be another point
2
2
∴
if 0

SLIDE 11

Pseudoinverse

Picture:

SLIDE 12

Method 1: Normal Equations

Solve
Compute Cholesky factorization

– i.e., , lower ∆

Compute by forward and backward solves
Complexity:

– ~ , ~ /3 ∴ ~ /3

SLIDE 13

Method 2: QR Factorization

Def:

is orthogonal if

– i.e.,
Theorem:
Proof:

SLIDE 14

Orthogonal Q

Note: multiplication by Q
Picture:

SLIDE 15

QR Factorization (reduced version)

Let
. Want to find orthonormal

vectors

such that

, … , , … , 1,2, … ,

This amounts to:
Matrix form:

– has orthonormal columns, is upper ∆

SLIDE 16

QR Factorization (full version)

Append additional
rthogonal cols to

– i.e., … ≡

Then
Usually for theoretical purpose

SLIDE 17

QR Factorization

Theorem: Suppose

has full rank.

unique orthogonal matrix

a unique upper matrix

with positive

diagonals ( such that

Picture:
Note: Cols of

are orthogonal to each other and their norm 1

SLIDE 18

QR Factorization

Consider the LS problem:
Then
Note:
CS475/CS675 (c) 2016 P. Poupart & J. Wan

SLIDE 19

QR Factorization

Note:
Proof:
(since
)
Picture:

SLIDE 20

QR Factorization

Pythagoras theorem:
The RHS is minimized if the first term is 0

– i.e., ⟹

Notes

⟺

CS475/CS675 (c) 2016 P. Poupart & J. Wan