regression via iteratively reweighted least squares alina
play

Regression via Iteratively Reweighted Least Squares Alina Ene, - PowerPoint PPT Presentation

Jun 25, 2023 •565 likes •1.02k views

Improved Convergence for and 1 Regression via Iteratively Reweighted Least Squares Alina Ene, Adrian Vladu IRLS Method Basic primitive: min r i x i 2 Ax = b IRLS Method Basic primitive: min r i x i 2 Ax = b solution

  1. Improved Convergence for ℓ ∞ and ℓ 1 Regression via Iteratively Reweighted Least Squares Alina Ene, Adrian Vladu

  2. IRLS Method Basic primitive: min ∑r i x i 2 Ax = b

  3. IRLS Method Basic primitive: min ∑r i x i 2 Ax = b solution given by one linear system solve x = R -1 A T (A T R -1 A) -1 Ab * R = diag(r)

  4. IRLS Method “Hard” problem: Basic primitive: ** min ∑r i x i 2 min |x| p Ax = b Ax = b solution given by one linear system solve * x = R -1 A T (A T R -1 A) -1 Ab * R = diag(r) ** p = {1, ∞ }

  5. IRLS Method “Hard” problem: Basic primitive: ** min ∑r i x i 2 min |x| p Ax = b Ax = b solution given by one equivalent to linear linear system solve programming * x = R -1 A T (A T R -1 A) -1 Ab * R = diag(r) ** p = {1, ∞ }

  6. IRLS Method “Hard” problem: Basic primitive: ** min ∑r i x i 2 min |x| p Ax = b Ax = b solution given by one equivalent to linear linear system solve programming * x = R -1 A T (A T R -1 A) -1 Ab * R = diag(r) ** p = {1, ∞ }

  7. IRLS Method “Hard” problem: Basic primitive: ** min ∑r i x i 2 min |x| p Ax = b Ax = b solution given by one equivalent to linear linear system solve programming * x = R -1 A T (A T R -1 A) -1 Ab * R = diag(r) ** p = {1, ∞ }

  8. IRLS Method “Hard” problem: Basic primitive: ** min ∑r i x i 2 min |x| p Ax = b Ax = b solution given by one equivalent to linear linear system solve programming * x = R -1 A T (A T R -1 A) -1 Ab * R = diag(r) ** p = {1, ∞ }

  9. IRLS Method “Hard” problem: Basic primitive: ** min ∑r i x i 2 min |x| p Ax = b Ax = b solution given by one equivalent to linear linear system solve programming * x = R -1 A T (A T R -1 A) -1 Ab * R = diag(r) ** p = {1, ∞ }

  10. Benchmark: Optimization on Graphs t min |x| ∞ s Ax = b

  11. Benchmark: Optimization on Graphs minimize congestion of flow x t min |x| ∞ s Ax = b

  12. Benchmark: Optimization on Graphs minimize congestion of flow x t min |x| ∞ s Ax = b boundary condition: x routes demand from s to t

  13. Benchmark: Optimization on Graphs minimize congestion of .5 .5 flow x .5 t 0 min |x| ∞ s Ax = b .5 .5 .5 boundary condition: x routes demand Maximum flow from s to t

  14. Benchmark: Optimization on Graphs +1 -1 min |x| 1 +1 Ax = b -1

  15. Benchmark: Optimization on Graphs minimize +1 cost of flow x -1 min |x| 1 +1 Ax = b -1

  16. Benchmark: Optimization on Graphs minimize +1 cost of flow x -1 min |x| 1 +1 Ax = b -1 boundary condition: x routes demand from +1 to -1

  17. Benchmark: Optimization on Graphs minimize +1 cost of 1 1 flow x 0 -1 0 min |x| 1 +1 Ax = b 0 1 1 -1 boundary condition: x routes demand Minimum cost flow from +1 to -1

  18. Benchmark: Optimization on Graphs min |x| 1 min |x| ∞ Ax = b Ax = b max flow min cost flow

  19. Benchmark: Optimization on Graphs min |x| 1 min |x| ∞ Q: Are these problems really that hard? Ax = b Ax = b max flow min cost flow

  20. Benchmark: Optimization on Graphs min |x| 1 min |x| ∞ Q: Are these problems really that hard? Ax = b Ax = b max flow min cost flow First order methods (gradient descent) ➜ running time strongly depends on matrix structure ➜ in general, takes time at least Ω(m 1.5 /poly(ε)) Second order methods (Newton method, IRLS) ➜ interior point method: Õ(m 1/2 ) linear system solves ➜ can be made Õ(n 1/2 ) with a lot of work [LS ’ 14] “Hybrid” method ➜ [CKMST, STOC ’ 11] Õ(m 1/3 /ε 11/3 ) linear system solves ➜ ~30 pages of description and proofs for complicated method

  21. Benchmark: Optimization on Graphs min |x| 1 min |x| ∞ Q: Are these problems really that hard? Ax = b Ax = b max flow min cost flow First order methods (gradient descent) ➜ running time strongly depends on matrix structure ➜ in general, takes time at least Ω(m 1.5 /poly(ε)) Second order methods (Newton method, IRLS) ➜ interior point method: Õ(m 1/2 ) linear system solves ➜ can be made Õ(n 1/2 ) with a lot of work [Lee-Sidford ’ 14] “Hybrid” method ➜ [CKMST, STOC ’ 11] Õ(m 1/3 /ε 11/3 ) linear system solves ➜ ~30 pages of description and proofs for complicated method

  22. Benchmark: Optimization on Graphs min |x| 1 min |x| ∞ Q: Are these problems really that hard? Ax = b Ax = b max flow min cost flow First order methods (gradient descent) ➜ running time strongly depends on matrix structure ➜ in general, takes time at least Ω(m 1.5 /poly(ε)) Second order methods (Newton method, IRLS) ➜ interior point method: Õ(m 1/2 ) linear system solves ➜ can be made Õ(n 1/2 ) with a lot of work [Lee-Sidford ’ 14] “Hybrid” method ➜ [Christiano-Kelner-Madry-Spielman-Teng ’ 11] Õ(m 1/3 /ε 11/3 ) linear system solves ➜ ~30 pages of description and proofs for complicated method

  23. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations

  24. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations * no matter what the structure of the underlying matrix is

  25. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations min |x| ∞ ≤ OPT Ax = b t s

  26. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations min |x| ∞ Guess ≤ OPT Ax = b OPT value (.5) t s

  27. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations min |x| ∞ Guess ≤ OPT Ax = b OPT value (.5) t s

  28. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations 1 min |x| ∞ Guess ≤ OPT 1 Ax = b 1 OPT value (.5) t r = 1 s Initialize 1 1 1

  29. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations 1 min |x| ∞ Guess ≤ OPT 1 .4 Ax = b 1 OPT value (.5) .4 t .6 .2 r = 1 s Initialize .6 .4 1 .4 min ∑ r i x i 2 1 Solve least 1 Ax = b squares problem

  30. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations 1 min |x| ∞ Guess ≤ OPT 1 .4 Ax = b 1.44 OPT value (.5) .4 t .6 .2 r = 1 s Initialize .6 .4 .4 1.44 min ∑ r i x i 2 1 Solve least 1 Ax = b squares problem r i ← r i * Update r max{(x i /OPT) 2 , 1}

  31. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations 1 min |x| ∞ Guess ≤ OPT 1 .4 Ax = b 1.44 OPT value (.5) .4 t .6 .2 r = 1 s Initialize .6 .4 .4 1.44 min ∑ r i x i 2 1 Solve least 1 Ax = b squares problem r i ← r i * Update r max{(x i /OPT) 2 , 1}

  32. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations 1 min |x| ∞ Guess ≤ OPT 1 Ax = b 1.44 OPT value (.5) t r = 1 s Initialize 1.44 min ∑ r i x i 2 1 Solve least 1 Ax = b squares problem r i ← r i * Update r max{(x i /OPT) 2 , 1}

  33. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations 1 min |x| ∞ Guess ≤ OPT 1 .44 Ax = b 1.44 OPT value (.5) .44 t .55 .11 r = 1 s Initialize .55 .44 .44 1.44 min ∑ r i x i 2 1 Solve least 1 Ax = b squares problem r i ← r i * Update r max{(x i /OPT) 2 , 1}

  34. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations 1 min |x| ∞ Guess ≤ OPT 1 .44 Ax = b 1.75 OPT value (.5) .44 t .55 .11 r = 1 s Initialize .55 .44 .44 1.75 min ∑ r i x i 2 1 Solve least 1 Ax = b squares problem r i ← r i * Update r max{(x i /OPT) 2 , 1}

  35. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations 1 min |x| ∞ Guess ≤ OPT 1 Ax = b 1.75 OPT value (.5) t r = 1 s Initialize 1.75 min ∑ r i x i 2 1 Solve least 1 Ax = b squares problem r i ← r i * Update r max{(x i /OPT) 2 , 1}

  36. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations min |x| ∞ Guess ≤ OPT Ax = b OPT value (.5) t r = 1 s Initialize min ∑ r i x i 2 Solve least Ax = b squares problem r i ← r i * Update r max{(x i /OPT) 2 , 1}

  37. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations 1 min |x| ∞ Guess ≤ OPT 1 2 Ax = b OPT value (.5) t r = 1 s Initialize 2 min ∑ r i x i 2 1 Solve least 1 Ax = b squares problem r i ← r i * Update r max{(x i /OPT) 2 , 1}

  38. This work Natural IRLS method runs in Õ(m 1/3 /ε 2/3 +1/ε 2 ) iterations 1 min |x| ∞ Guess ≤ OPT 1 .5 2 Ax = b OPT value .5 (.5) t .5 0 r = 1 s Initialize .5 .5 2 .5 min ∑ r i x i 2 1 Solve least 1 Ax = b squares problem r i ← r i * Update r max{(x i /OPT) 2 , 1}

  39. Nonstandard Optimization Primitive ➜ Objective function is max r≥0 min Ax=b ∑r i x i 2 /∑r i Similar analysis to packing/covering LP [Young ’ 01] ℓ 1 version is a type of “slime mold dynamics” [Straszak- Vishnoi ’ 16, ‘17]

  40. Nonstandard Optimization Primitive ➜ Objective function is max r≥0 min Ax=b ∑r i x i 2 /∑r i ➜ Similar analysis to packing/covering LP [Young ’ 01] ℓ 1 version is a type of “slime mold dynamics” [Straszak- Vishnoi ’ 16, ‘17]

  41. Nonstandard Optimization Primitive ➜ Objective function is max r≥0 min Ax=b ∑r i x i 2 /∑r i ➜ Similar analysis to packing/covering LP [Young ’ 01] ➜ ℓ 1 version is a type of “slime mold dynamics” [Straszak- Vishnoi ’ 16, ‘17]

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Iteratively reweighted penalty alternating minimization methods with continuation for image

Iteratively reweighted penalty alternating minimization methods with continuation for image

Iteratively reweighted penalty alternating minimization methods with continuation for image deblurring Tao Sun, Dongsheng Li, Hao Jiang, Zhe Quan NUDT& HNU Tao Sun, Dongsheng Li, Hao Jiang, Zhe Quan Iteratively reweighted penalty

275 views • 12 slides
Iteratively Reweighted 1 Approaches to Sparse Composite Regularization Phil Schniter Joint

Iteratively Reweighted 1 Approaches to Sparse Composite Regularization Phil Schniter Joint

Iteratively Reweighted 1 Approaches to Sparse Composite Regularization Phil Schniter Joint work with Prof. Rizwan Ahmad (OSU) Supported in part by NSF grant CCF-1018368. MATHEON Conf. on Compressed Sensing and its Applications TU-Berlin

308 views • 29 slides
1 Least Squares Regression Suppose someone hands you a stack of N vectors, { x N } , each of

1 Least Squares Regression Suppose someone hands you a stack of N vectors, { x N } , each of

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 3B notes: Least Squares Regression 1 Least Squares Regression Suppose someone hands you a stack of N vectors, { x N } , each

291 views • 4 slides
Least Mean Squares Regression Machine Learning 1 Least Squares Method for regression

Least Mean Squares Regression Machine Learning 1 Least Squares Method for regression

Least Mean Squares Regression Machine Learning 1 Least Squares Method for regression Examples The LMS objective Gradient descent Incremental/stochastic gradient descent 2 Least Squares Method for regression Examples The

1.23k views • 53 slides
Deep Learning - Theory and Practice Linear Regression, Least Squares 20-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 20-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 20-02-2020 Classification and Logistic Regression http://leap.ee.iisc.ac.in/sriram/teaching/DL20/ deeplearning.cce2020@gmail.com Least Squares for Classification K-class

467 views • 24 slides
Least Squares (outline) Standard regression: Fit data with weighted sum of regressors.

Least Squares (outline) Standard regression: Fit data with weighted sum of regressors.

Mathematical Tools for Neural and Cognitive Science Fall semester, 2018 Section 2: Least Squares Least Squares (outline) Standard regression: Fit data with weighted sum of regressors. Solution via calculus, orthogonality, SVD

377 views • 9 slides
Weighted Least Squares Recall the linear regression equation E ( Y ) = 0 + 1 x 1 + 2 x 2

Weighted Least Squares Recall the linear regression equation E ( Y ) = 0 + 1 x 1 + 2 x 2

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II Weighted Least Squares Recall the linear regression equation E ( Y ) = 0 + 1 x 1 + 2 x 2 + + k x k We have estimated the

680 views • 11 slides
Deep Learning - Theory and Practice Linear Regression, Least Squares 13-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 13-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 13-02-2020 Classification and Logistic Regression http://leap.ee.iisc.ac.in/sriram/teaching/DL20/ deeplearning.cce2020@gmail.com Linear Regression Solution to Maximum

145 views • 11 slides
Business Statistics CONTENTS Ordinary least squares (recap for some) Statistical formulation of

Business Statistics CONTENTS Ordinary least squares (recap for some) Statistical formulation of

SIMPLE REGRESSION ANALYSIS Business Statistics CONTENTS Ordinary least squares (recap for some) Statistical formulation of the regression model Assessing the regression model Testing the regression coefficients The ANOVA table Old exam

554 views • 30 slides
Deep Learning - Theory and Practice Linear Regression, Least Squares 27-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 27-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 27-02-2020 Classification and Logistic Regression http://leap.ee.iisc.ac.in/sriram/teaching/DL20/ deeplearning.cce2020@gmail.com Logistic Regression 2- class logistic

308 views • 15 slides
10-601 Machine Learning Regression Outline Regression vs Classification Linear regression

10-601 Machine Learning Regression Outline Regression vs Classification Linear regression

10-601 Machine Learning Regression Outline Regression vs Classification Linear regression another discriminative learning method As optimization Gradient descent As matrix inversion (Ordinary Least Squares) Overfitting

1.52k views • 97 slides
of the cytostatic bosutinib by nonlinear least-squares regression of multiwavelength

of the cytostatic bosutinib by nonlinear least-squares regression of multiwavelength

The dissociation constants of the cytostatic bosutinib by nonlinear least-squares regression of multiwavelength spectrophotometric and potentiometric pH-titration data * Milan Meloun 1 , Veronika Neasov 1 , Milan Javrek 2 and Tom

204 views • 9 slides
The Simple Regression Model Deriving the Ordinary Least Squares Estimates Properties of Caio

The Simple Regression Model Deriving the Ordinary Least Squares Estimates Properties of Caio

Definition of the Simple Regression Model The Simple Regression Model Deriving the Ordinary Least Squares Estimates Properties of Caio Vigo OLS on any Sample of Data The University of Kansas Units of Measurement Department of

999 views • 83 slides
The Simple Regression Model Deriving the Ordinary Least Squares Estimates Properties of Caio

The Simple Regression Model Deriving the Ordinary Least Squares Estimates Properties of Caio

Definition of the Simple Regression Model The Simple Regression Model Deriving the Ordinary Least Squares Estimates Properties of Caio Vigo OLS on any Sample of Data The University of Kansas Units of Measurement Department of

1.21k views • 86 slides
w o o o o o o o x o o o x o o o that represents how aligned the x x x x x x

w o o o o o o o x o o o x o o o that represents how aligned the x x x x x x

Outline Logistic function IAML: Logistic Regression Logistic regression Learning logistic regression Optimization Nigel Goddard The power of non-linear basis functions School of Informatics Least-squares classification

622 views • 6 slides
Chapter 5 Least Squares Chapter 5 Inconsistent Systems In regression (and many other

Chapter 5 Least Squares Chapter 5 Inconsistent Systems In regression (and many other

Chapter 5 Least Squares Chapter 5 Inconsistent Systems In regression (and many other applications) we have a system of equations wed like to solve: X = y However, this system does not have an exact solution. (i.e. all of our data points

386 views • 11 slides
On Approximating the Covering Radius and Finding Dense Lattice Subspaces Daniel Dadush Centrum

On Approximating the Covering Radius and Finding Dense Lattice Subspaces Daniel Dadush Centrum

On Approximating the Covering Radius and Finding Dense Lattice Subspaces Daniel Dadush Centrum Wiskunde & Informatica (CWI) ICERM April 2018 Outline 1. Integer Programming and the Kannan-Lovsz (KL) Conjecture. 2. 2 KL Conjecture

1.13k views • 69 slides
Hypergeometric L -functions in average polynomial time Edgar Costa, Kiran S. Kedlaya, and David

Hypergeometric L -functions in average polynomial time Edgar Costa, Kiran S. Kedlaya, and David

Hypergeometric L -functions in average polynomial time Edgar Costa, Kiran S. Kedlaya, and David Roe Costa, Roe: Department of Mathematics, Massachusetts Institute of Technology Kedlaya: Department of Mathematics, University of California, San

388 views • 18 slides
UNDERSTANDING PROGRAM EFFICIENCY: 1 (download slides and .py files and follow along!) 6.0001

UNDERSTANDING PROGRAM EFFICIENCY: 1 (download slides and .py files and follow along!) 6.0001

UNDERSTANDING PROGRAM EFFICIENCY: 1 (download slides and .py files and follow along!) 6.0001 LECTURE 10 1 6.0001 LECTURE 10 Today Measuring orders of growth of algorithms Big Oh notaAon Complexity classes 6.0001 LECTURE 10 2

410 views • 39 slides
From Ramsey to Ehrenfeucht: a reduction between games Oleg Verbitsky Humboldt Universit at

From Ramsey to Ehrenfeucht: a reduction between games Oleg Verbitsky Humboldt Universit at

From Ramsey to Ehrenfeucht: a reduction between games Oleg Verbitsky Humboldt Universit at IAPMM and Berlin, Germany Lviv, Ukraine Bertinoro, October 2009 Joint work with Frank Harary and Wolfgang Slany. Basics of Graph Ramsey theory

481 views • 22 slides
UMBC A N R Y L D A B M A L F T U M B C I O M 1 (March 29, 1998 4:05 pm) Y O

UMBC A N R Y L D A B M A L F T U M B C I O M 1 (March 29, 1998 4:05 pm) Y O

Principles of VLSI Design VHDL CMSC 491B/711 VHDL Introduction A language for describing the structural, physical and behavioral characteristics of digital systems. Execution of a VHDL program results in a simulation of the digital system.

719 views • 23 slides
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao

Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao

Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks Yuan Cao and Quanquan Gu Computer Science Department 1 / 14 Learning Over-parameterized DNNs Empirical observation on extremely wide deep neural networks

631 views • 14 slides
Dependency Parsing Joakim Nivre Uppsala University Department of Linguistics and Philology

Dependency Parsing Joakim Nivre Uppsala University Department of Linguistics and Philology

Dependency Parsing Joakim Nivre Uppsala University Department of Linguistics and Philology joakim.nivre@lingfil.uu.se Dependency Parsing 1(14) 1. Dependency Trees 2. Arc-Factored Models 3. Online Learning 4. Eisners Algorithm 5.

726 views • 14 slides
Improving Your TABLEGEN Description Javed Absar WHAT IS TABLEGEN ? DSL invented for LLVM

Improving Your TABLEGEN Description Javed Absar WHAT IS TABLEGEN ? DSL invented for LLVM

Improving Your TABLEGEN Description Javed Absar WHAT IS TABLEGEN ? DSL invented for LLVM Used extensively Describe records of Instructions, registers, intrinsics, attributes, scheduler model, warnings, 2

412 views • 17 slides

More recommend