CS475 / CS675 Lecture 10: June 2, 2016 Least Squares Problems - PowerPoint PPT Presentation

CS475 / CS675 Lecture 10: June 2, 2016 Least Squares Problems Reading: [TB] Chapt 11 CS475/CS675 (c) 2016 P. Poupart & J. Wan 1

Least Squares Problems • First posed and formulated by Gauss. • Surveyors tried to identify boundaries by measuring certain angles and distances from known landmarks. • To update the location of landmarks, new measurements of angles and distances between landmarks are made. CS475/CS675 (c) 2016 P. Poupart & J. Wan 2

Surveying Example • Given a set of old locations , find correction � � such that better � � � � � � match new measurements • Picture: CS475/CS675 (c) 2016 P. Poupart & J. Wan 3

Surveying Example • Non‐linear constraint: cos � � � � ����������� � � � � �� � � � �� � � � � � �� � � � �� � � � � �� � � � �� � • Suppose � � � � – Multiply through the denominator – Multiply out all the terms to get a quartic polynomial in all � ‐variables – Throw away all terms containing � � , � � , � � ⟹ linear constraint CS475/CS675 (c) 2016 P. Poupart & J. Wan 4

Surveying example • Collect all linear constraints for all angles and distance measurements ⟹ overdetermined linear system • In general: � � � � � � �� � � � ⋯ � � �� � � � � � ⋮ constraints observations � �� � � � � �� � � � ⋯ � � �� � � � � � Picture: �� � � CS475/CS675 (c) 2016 P. Poupart & J. Wan 5

Overdetermined system • In general: • Idea: minimize the residual vector – Optimization problem: • Least squares (LS) problems CS475/CS675 (c) 2016 P. Poupart & J. Wan 6

Solving LS Problems • Geometric interpretation: CS475/CS675 (c) 2016 P. Poupart & J. Wan 7

Solving LS Problems ��� , � , • Theorem: Let . A vector � minimizes � � � � if and only if � � • � � � CS475/CS675 (c) 2016 P. Poupart & J. Wan 8

Pseudoinverse � is called � � �� • Def: the pseudoinverse of • The least squares solution is given by � �� � � � ? • Why is the minimizer of CS475/CS675 (c) 2016 P. Poupart & J. Wan 9

Pseudoinverse � • Let be another point � � � � �� � � � � �� � � � �� � � � � � �� � �� � � � �� � �� � � � �� � � � �� � 2 �� � � � �� � �� � �� � � � � 2� � � � � � � � �� � � � �� �� � � � � � � �� �� � � � � � �� � ∴ � � �� if � � 0 CS475/CS675 (c) 2016 P. Poupart & J. Wan 10

Pseudoinverse • Picture: CS475/CS675 (c) 2016 P. Poupart & J. Wan 11

Method 1: Normal Equations � � • Solve • Compute Cholesky factorization – i.e., � � � � �� � , � � lower ∆ � � • Compute by forward and backward solves • Complexity: – ����� � � � ~ �� � , ����� �� � ~ � � /3 ∴ ����� ����� ~ �� � � � � /3 �� � �� CS475/CS675 (c) 2016 P. Poupart & J. Wan 12

Method 2: QR Factorization �� � • Def: is orthogonal if – i.e., � � � � �� � � � • Theorem: � � • Proof: CS475/CS675 (c) 2016 P. Poupart & J. Wan 13

Orthogonal Q • Note: multiplication by Q • Picture: CS475/CS675 (c) 2016 P. Poupart & J. Wan 14

QR Factorization (reduced version) • Let � . Want to find orthonormal � � vectors � such that ���� � � , … , � � � ������ � , … , � � � � � 1,2, … , � • This amounts to: • Matrix form: � has orthonormal columns, � � is upper ∆ – � CS475/CS675 (c) 2016 P. Poupart & J. Wan 15

QR Factorization (full version) • Append additional orthogonal cols to � ��� – i.e., �� ��� � ��� … � � � ≡ � • Then • Usually for theoretical purpose CS475/CS675 (c) 2016 P. Poupart & J. Wan 16

QR Factorization ��� has full rank. • Theorem: Suppose ��� � unique orthogonal matrix and ��� with positive a unique upper matrix diagonals ( �� such that • Picture: • Note: Cols of are orthogonal to each other and their norm 1 CS475/CS675 (c) 2016 P. Poupart & J. Wan 17

QR Factorization • Consider the LS problem: � � • Then � � � � � � • Note: CS475/CS675 (c) 2016 P. Poupart & J. Wan 18

QR Factorization � � • Note: � � � � � � � � � �� � � �� �� • Proof: � � � � � � � � � � � �� � � �� �� � � � � � � � � � � � �� � � �� � � � �� � � � � � � � � ) � 0 (since � • Picture: CS475/CS675 (c) 2016 P. Poupart & J. Wan 19

QR Factorization • Pythagoras theorem: � � � � � � � � � �� � � � � � �� �� � � � � � � � � � � � � � � � �� �� � � � � � � � � • The RHS is minimized if the first term is 0 �� � � � � � ⟹ � � � � �� � � � � – i.e., � • Notes � � � � � �� � � � 1. � � � � � � � � �� � � � � � � � � � �� � � � � ⟺ � � 2. � � � �� � � � � � � � � � � � � �� � � � � �� � � � � � � � CS475/CS675 (c) 2016 P. Poupart & J. Wan 20