Practical Least-Squares for Computer Graphics Siggraph Course 11 - PowerPoint PPT Presentation

Outline Least Squares with Generalized Errors Robust Least Squares Constrained Least Squares Practical Least-Squares for Computer Graphics Siggraph Course 11 Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Constrained Least Squares Least Squares with Generalized Errors 1 Weighted Least Squares Total Least Squares Robust Least Squares 2 Motivation Non-Gaussian Distributions Robust error measures Iteratively Reweighted Least Squares Motivation Motivation Least Median of Squares Constrained Least Squares 3 Lagrange Multipliers Non-Negative Least Squares Inequality Constraints Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Sermon Motivation moment Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Weighted Least Squares Extend least squares to account for data with different noise variance per-sample, or missing data � 2 � ∑ m j = 1 A i , j x j − b i n ∑ argmin . σ 2 x i = 1 i Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Weighted Least Squares � 2 � ∑ m j = 1 A i , j x j − b i n ∑ argmin . σ 2 x i = 1 i rewrite in matrix terms with W being a diagonal matrix W ii = 1 σ i x ( W ( b − Ax )) T ( W ( b − Ax )) , ⇒ argmin Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Weighted Least Squares rewrite once more x ( W ( b − Ax )) T ( W ( b − Ax )) , argmin x ( b − Ax ) T W T W ( b − Ax )) ⇒ argmin Rule: ( ab ) T = b T a T Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Weighted Least Squares Big picture: x ( b − Ax ) T W T W ( b − Ax )) , argmin This is a “scalar” (a single number), expressing the summed weighted error. Take the derivative with respect to x and set to zero. The solve for x . Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Matrix calculus I derivative of scalar w.r.t scalar is scalar derivative of scalar w.r.t vector is vector derivative of scalar w.r.t matrix is matrix d x = [ ds ds , ds , ds , ··· ] dx 1 dx 2 dx 3 d d xx T Ax = 2 Ax Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Matrix calculus II “scalar” matrix dx x 2 ⇒ 2 x d dx x T x ⇒ 2 x d dx ax 2 ⇒ 2 ax d dx x T Ax ⇒ 2 Ax d A symmetric d d dx a T x ⇒ a dx ax ⇒ a Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Weighted Least Squares Goal: x ( b − Ax ) T W T W ( b − Ax )) argmin Expand b T W 2 ( b − Ax ) − x T A T W 2 ( b − Ax ) b T W 2 b − b T W 2 Ax − x T A T W 2 b + x T A T W 2 Ax Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Weighted Least Squares b T W 2 b − b T W 2 Ax − x T A T W 2 b + x T A T W 2 Ax ⇒ b T W 2 b − 2 b T W 2 Ax + x T A T W 2 Ax x T A T W 2 b is a scalar, legal to “transpose” a scalar. Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Weighted Least Squares ax 2 ” “ c + bx + b T W 2 b − 2 b T W 2 Ax + x T A T W 2 Ax d d x = 0 − 2 A T W 2 b + 2 A T W 2 Ax = 0 Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Weighted Least Squares d d x = 0 − 2 A T W 2 b + 2 A T W 2 Ax = 0 A T W 2 Ax = A T W 2 b x = ( A T W 2 A ) − 1 A T W 2 b = 0 Although A may not be square, A T W 2 A will be Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Total Least Squares Total Least Squares measures closest error to the (line), rather than in the y direction. Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Total Least Squares Unusual: A least squares problem formulation leads to an eigenvalue problem rather than a linear system! Also requires Lagrange multiplers (constrained LS section...). Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Total Least Squares: Applications Surface fitting. N. Amenta and Y. J. Kil. Defining point-set surfaces, SIGGRAPH 2004. Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Weighted Least Squares Constrained Least Squares Total Least Squares Reminders Win a high-end graphics card (HD2900XT) by filling out the course evaluation: http://www.siggraph.org/courses evaluation Course web site (download corrected slides after course): http://www.siggraph.org/courses evaluation Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Motivation Constrained Least Squares Non-Gaussian Distributions Robust error measures Robust: Outline Motivation Redescending error measures Iteratively reweighted least squares (IRLS) RANSAC Least Median of Squares Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Motivation Constrained Least Squares Non-Gaussian Distributions Robust error measures Robust Least Squares: Motivation Even a single accidental point (outlier, red point) can destroy an “ordinary” least squares fit. Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Motivation Constrained Least Squares Non-Gaussian Distributions Robust error measures Robust Least Squares: Applications M. Black thesis: introduced robust statistics in optic flow. Application: optic flow-based face tracking on the Matrix sequels Borshukov et al. Universal Capture - Image-based Facial Animation for “The Matrix Reloaded” Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Motivation Constrained Least Squares Non-Gaussian Distributions Robust error measures Non-Gaussian Distributions A high kurtosis density (heavy line) has both more data close to the mean, and more outliers, than a Gaussian distribution (light line). Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Motivation Constrained Least Squares Non-Gaussian Distributions Robust error measures Appropriateness of Gaussian Gaussian distribution is appropriate when a large number of independent effects are summed (stock market): The distribution of a sum is the convolution of the individual distributions. Multiple convolution rapidly converges to Gaussian. Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Motivation Constrained Least Squares Non-Gaussian Distributions Robust error measures Appropriateness of Gaussian Gaussian distribution is not necessarily appropriate when the error is due to a single cause, a few large isolate events, or when the distribution is otherwise simply “non Gaussian”. Siggraph Course 11 Practical Least-Squares for Computer Graphics

Outline Least Squares with Generalized Errors Robust Least Squares Motivation Constrained Least Squares Non-Gaussian Distributions Robust error measures Robust error measures 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0. 1 0 −1 − 0.8 − 0.6 − 0.4 − 0.2 0 0.2 0.4 0.6 0.8 1 � x � � 2 � 1 + 1 The redescending estimator function log (red) 2 σ versus the standard quadratic error y = x 2 (blue). Siggraph Course 11 Practical Least-Squares for Computer Graphics