Nonconvex Variance Reduced Optimization with Arbitrary Sampling - PowerPoint PPT Presentation

Nov 09, 2022 •780 likes •1.07k views

Nonconvex Variance Reduced Optimization with Arbitrary Sampling Samuel Horvth Peter Richtrik Empirical Risk Minimization n x R d f ( x ) := 1 X min f i ( x ) n i =1 Empirical Risk Minimization n x R d f ( x ) := 1 X min f i (

Nonconvex Variance Reduced Optimization with Arbitrary Sampling Samuel Horváth Peter Richtárik
Empirical Risk Minimization n x ∈ R d f ( x ) := 1 X min f i ( x ) n i =1
Empirical Risk Minimization n x ∈ R d f ( x ) := 1 X min f i ( x ) n i =1 n is big
Empirical Risk Minimization n x ∈ R d f ( x ) := 1 X min f i ( x ) n i =1 n is big non-convex, L i -smooth kr f i ( x ) � r f i ( y ) k  L i k x � y k
Baseline Variance Reduced SGD Methods SVRG Johnson & Zhang NIPS 2013 SAGA Defazio, Bach & Lacoste-Julien NIPS 2014 SARAH Nguyen, Liu, Scheinberg & Takáč ICML 2017
Baseline Variance Reduced SGD Methods SVRG Johnson & Zhang NIPS 2013 Uniform sampling SAGA Defazio, Bach & Lacoste-Julien NIPS 2014 Uniform sampling SARAH Nguyen, Liu, Scheinberg & Takáč ICML 2017 Uniform sampling
Baseline Variance Reduced SGD Methods–Mini-batch SVRG Konečný & Richtárik FAMS 2017 SAGA Reddi, Hefny, Sra, Poczos, Smola CDC 2016 SARAH Nguyen, Liu, Scheinberg & Takáč 2017
Baseline Variance Reduced SGD Methods–Mini-batch Mini-batch size SVRG Konečný & Richtárik FAMS 2017 Uniform sampling SAGA Reddi, Hefny, Sra, Poczos, Smola CDC 2016 Uniform sampling SARAH Nguyen, Liu, Scheinberg & Takáč 2017 Uniform sampling
Contributions Analysis of SVRG, SAGA and SARAH in the arbitrary sampling paradigm • Construction of optimal minibatch sampling •
Richtárik & Takáč (OL 2016; arXiv 2013) Contributions Qu, Richtárik & Zhang (NIPS 2015) Qu & Richtárik (COAP 2016) Chambolle, Ehrhardt, Richtárik & Schoenlieb (SIOPT 2018) Hanzely & Richtárik (AISTATS 2019) Qian, Qu & Richtárik (ICML 2019) Gower, Loizou, Qian, Sailanbayev, Shulgin & Richtárik (ICML 2019) Analysis of SVRG, SAGA and SARAH in the arbitrary sampling paradigm • Construction of optimal minibatch sampling •
Richtárik & Takáč (OL 2016; arXiv 2013) Contributions Qu, Richtárik & Zhang (NIPS 2015) Qu & Richtárik (COAP 2016) Chambolle, Ehrhardt, Richtárik & Schoenlieb (SIOPT 2018) Hanzely & Richtárik (AISTATS 2019) Qian, Qu & Richtárik (ICML 2019) Gower, Loizou, Qian, Sailanbayev, Shulgin & Richtárik (ICML 2019) Analysis of SVRG, SAGA and SARAH in the arbitrary sampling paradigm • Construction of optimal minibatch sampling • First optimal/importance sampling for minibatches!

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization Kaiyi Ji

Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization Kaiyi Ji 1 , Zhe Wang 1 , Yi Zhou 2 , Yingbin Liang 1 1 Ohio State University, 2 Duke University ICML 2019 K. Ji, Z. Wang, Y. Zhou, Y. Liang Zeroth-Order

327 views • 9 slides

Stochastic Cubic Regularization for Fast Nonconvex Optimization Nilesh Tripuraneni, Mitchell

Stochastic Cubic Regularization for Fast Nonconvex Optimization Nilesh Tripuraneni, Mitchell Stern, Chi Jin, Jeffrey Regier and Michael I. Jordan Achin Jain University of Pennsylvania STAT991, Spring 2019 Fast Nonconvex Optimization 1

422 views • 26 slides

Estimate Sequences for Variance-Reduced Stochastic Composite Optimization Andrei Kulunchakov

Estimate Sequences for Variance-Reduced Stochastic Composite Optimization Andrei Kulunchakov Julien Mairal andrei.kulunchakov@inria.fr julien.mairal.@inria.fr International Conference on Machine Learning, 2019 Poster event-4062, (Jun 12th,

317 views • 6 slides

Pathwise Coordinate Optimization for Nonconvex Sparse Learning Tuo Zhao

Pathwise Coordinate Optimization for Nonconvex Sparse Learning Tuo Zhao http://www.princeton.edu/tuoz Department of Computer Science Johns Hopkins University Mar. 25. 2015 A General Theory of Pathwise Coordinate Optimization Collaborators

798 views • 45 slides

Accelerated Douglas-Rachford splitting and ADMM for structured nonconvex optimization Panos

Accelerated Douglas-Rachford splitting and ADMM for structured nonconvex optimization Panos Patrinos KU Leuven (ESAT-STADIUS) joint work with Andreas Themelis and Lorenzo Stella LCCC Workshop Large-Scale and Distributed Optimization Lund,

591 views • 47 slides

Capstone Technology Industrial Plant Optimization in Reduced Dimensional Spaces Fields

Capstone Technology Industrial Plant Optimization in Reduced Dimensional Spaces Fields Optimization Lecture Toronto, ON Giles Laurier June 4, 2013 Agenda Review of optimization in oil refining Real Time Optimization Reduced Space

845 views • 56 slides

How to Characterize the Worst-Case Performance of Algorithms for Nonconvex Optimization Frank E.

Motivation Contemporary Analyses Partitioning Regularization Methods Summary How to Characterize the Worst-Case Performance of Algorithms for Nonconvex Optimization Frank E. Curtis , Lehigh University joint work with Daniel P. Robinson ,

734 views • 37 slides

Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization Rie

Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization Rie Johnson and Tong Zhang RJ Research Consulting Hong Kong University of Science and Technology 1 / 12 Training Deep Neural Networks

458 views • 21 slides

Nonconvex Distributed Optimization: Novel Algorithmic Design and Arbitrarily Precise Solutions

Nonconvex Distributed Optimization: Novel Algorithmic Design and Arbitrarily Precise Solutions Presenter: Zhiyu He Coauthors: Jianping He*, Cailian Chen and Xinping Guan Department of Automation Shanghai Jiao Tong University July 2020

716 views • 39 slides

A Trust Funnel Algorithm for Nonconvex Equality Constrained Optimization with O ( 3 / 2 )

Motivation Proposed Algorithm Theoretical Results Numerical Results Summary A Trust Funnel Algorithm for Nonconvex Equality Constrained Optimization with O ( 3 / 2 ) Complexity Mohammadreza Samadi , Lehigh University joint work with

556 views • 39 slides

A Lagrangean Based Branch-and-Cut Algorithm for Global Optimization of Nonconvex Mixed-Integer

A Lagrangean Based Branch-and-Cut Algorithm for Global Optimization of Nonconvex Mixed-Integer Nonlinear Programs with Decomposable Structures Ramkumar Karuppiah and Ignacio E. Grossmann Department of Chemical Engineering Carnegie Mellon

588 views • 48 slides

Flexible ADMM for Block-Structured Convex and Nonconvex Optimization Zhi-Quan (Tom) Luo Joint

Introduction The ADMM Algorithm The Main Result Flexible ADMM for Block-Structured Convex and Nonconvex Optimization Zhi-Quan (Tom) Luo Joint work with Mingyi Hong, Tsung-Hui Chang, Xiangfeng Wang, Meisam Razaviyanyn, Shiqian Ma University

770 views • 57 slides

Interior point method for nonlinear nonconvex optimization Ladislav Lukan, Ctirad Matonoha, Jan

Interior point method for nonlinear nonconvex optimization Ladislav Lukan, Ctirad Matonoha, Jan Vl cek Institute of Computer Science AS CR, Prague GAMM Workshop Applied and Numerical Linear Algebra September 11-12, 2008 Technische

842 views • 51 slides

Faster Stochastic Alternating Direction Method of Multipliers for Nonconvex Optimization Feihu

Faster Stochastic Alternating Direction Method of Multipliers for Nonconvex Optimization Feihu Huang 1 , Songcan Chen 2,3 and Heng Huang 1,4* 1. Department of Electrical and Computer Engineering, University of Pittsburgh, USA 2. College of

179 views • 14 slides

Variance = E[I 2 ] 2pE[I] + p 2 = E[I] 2p p + p 2 = 2 2 = p-2p+ p pq variance.1

Variance of an Indicator Mathematics for Computer Science MIT 6.042J/18.062J IanindicatorwithE[I]=p: Var[I]:: = E[(I p) 2 ] Variance = E[I 2 ] 2pE[I] + p 2 = E[I] 2p p + p 2 = 2 2 = p-2p+ p pq variance.1 variance.2

104 views • 6 slides

Inexact variable metric proximal gradient methods with line-search for convex and nonconvex

Inexact variable metric proximal gradient methods with line-search for convex and nonconvex optimization Silvia Bonettini Dipartimento di Scienze Fisiche, Optimization Algorithms and Software Informatiche e Matematiche for Inverse problemS

567 views • 17 slides

Me vs BigQuery CEO @ Applications Databases Files Stripe Asana Instagram Amazon Aurora

Data Warehouse Benchmark: Redshift vs Snowflake Me vs BigQuery CEO @ Applications Databases Files Stripe Asana Instagram Amazon Aurora Amazon Cloudfront Xero Bing Ads Intercom Amazon RDS Amazon Kinesis Firehose Zendesk Braintree

718 views • 25 slides

Software patterns for productive teams Radoslav Georgiev, @Rado_g, EuroPython 2019 3rd EuroPython

Software patterns for productive teams Radoslav Georgiev, @Rado_g, EuroPython 2019 3rd EuroPython for me Goal of this talk: Be practical , pragmatic & provide value . Goal of this talk: Aha! We should try this moment. Context : Im

718 views • 44 slides

Differentiable Ranking and Sorting using Optimal Transport M. Cuturi O. Teboul J.-P. Vert

Differentiable Ranking and Sorting using Optimal Transport M. Cuturi O. Teboul J.-P. Vert google_research/soft_sort/ @ East Exhibition Hall B + C #60 <latexit

614 views • 27 slides

Drew Scott Unsworth King Global Head of Digital & Marketing C.O.O., Co-founder - Accordant

Drew Scott Unsworth King Global Head of Digital & Marketing C.O.O., Co-founder - Accordant Services - Xero General Manager, Digital Banking Solution Specialist, Personalisation - Commonwealth Bank - Adobe 2 DIGITAL TRANSFORMATION 3

717 views • 14 slides

Free and Low Cost Technology James Marshall Berry JMB Web Consulting Geni Whitehouse

7/10/2012 Free and Low Cost Technology James Marshall Berry JMB Web Consulting Geni Whitehouse Brotemarkle, Davis & Co. 1 1 July 10 11, 2012 Napa Valley Marriott Introductions James Marshall Berry @WinoTone After

330 views • 9 slides

Centrality Argimiro Arratia & R. Ferrer-i-Cancho Universitat Polit` ecnica de Catalunya Version

Graph-theoretical centrality Eigenvector-based centrality Miscellanea Centrality Argimiro Arratia & R. Ferrer-i-Cancho Universitat Polit` ecnica de Catalunya Version 0.4 Complex and Social Networks (20 20 -202 1 ) Master in Innovation and

567 views • 44 slides

Dynamical Systems and ADMM Ren Vidal Herschel Seder Professor of Biomedical Engineering

Dynamical Systems and ADMM Ren Vidal Herschel Seder Professor of Biomedical Engineering Director of the Mathematical Institute for Data Science Johns Hopkins University <latexit

536 views • 22 slides

Presented by: Peggy Hall, Legacy Accounting and Software Training Sponsored by: KCAA (Kitsap

Presented by: Peggy Hall, Legacy Accounting and Software Training Sponsored by: KCAA (Kitsap Community and Agricultural Alliance) WSU Regional Small Farms Program Useful Resources Start Up Decisions/Farm or Hobby? General Guidelines

605 views • 33 slides