Empirical Loss Minimization Traffic sign - STOP Sample - PowerPoint PPT Presentation

Aug 17, 2023 •333 likes •738 views

Empirical Loss Minimization Traffic sign - STOP Sample i.i.d. points Stochastic Gradient Descent Lon Bottou, Frank E Curtis, Jorge Nocedal Optimization methods for large-scale machine learning SVRG:

č
č
Empirical Loss Minimization
Traffic sign - STOP
Sample i.i.d. points
Stochastic Gradient Descent
● ● ● ● ● Léon Bottou, Frank E Curtis, Jorge Nocedal Optimization methods for large-scale machine learning
SVRG: Stochastic Variance Reduced Gradient
● Unbiased stochastic gradient:
● ●
●
SAG/SAGA
● ● ● ● ●
● ●
SARAH č
● ● ●
● ●
● ● ●
● ● …
RCV Dataset SVRG and SARAH need full gradient after restart Variance of SVRG is decreased after each restart Variance of SARAH goes to zero
SARAH+ Practical Variant
good performance across many datasets
Numerical Experiments
One has to tune parameters to get a good performance! Not for SARAH+!
Summary
● ● ●
Convex Case
Non-Convex Case
Any Questions?

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Di ff erentially Private Empirical Risk Minimization with Non-convex Loss Functions Di Wang ,

Di ff erentially Private Empirical Risk Minimization with Non-convex Loss Functions Di Wang , Changyou Chen and Jinhui Xu State University of New York at Bu ff alo International Conference on Machine Learning 2019 Di Wang Non-convex DP-ERM ICML

544 views • 27 slides

Learning as Loss Minimization Machine Learning 1 Learning as loss minimization The setup

Learning as Loss Minimization Machine Learning 1 Learning as loss minimization The setup Examples x drawn from a fixed, unknown distribution D Hidden oracle classifier f labels examples We wish to find a hypothesis h that mimics f

294 views • 25 slides

Empirical Risk Minimization October 29, 2015 Outline Empirical risk minimization view

Empirical Risk Minimization October 29, 2015 Outline Empirical risk minimization view Perceptron CRF Notation for Linear Models Training data: {(x 1 , y 1 ), (x 2 , y 2 ), , (x N , y N )} Testing data: {(x N+1 , y N+1 ),

418 views • 21 slides

CSC2412: Private Gradient Descent & Empirical Risk Minimization Sasho Nikolov 1 Empirical

CSC2412: Private Gradient Descent & Empirical Risk Minimization Sasho Nikolov 1 Empirical Risk Minimization Learning: Reminder Known data universe X and an unknown probability distribution D on X Known concept class C and an unknown

341 views • 17 slides

WEIGHTED SUMS OF RANDOM KITCHEN SINKS Replacing minimization with randomization in learning

WEIGHTED SUMS OF RANDOM KITCHEN SINKS Replacing minimization with randomization in learning The model Given a set of training data in a domain Fit a function to minimize risk Empirical Risk Risk Loss Function Hinge loss

621 views • 13 slides

2019-2020 What is a consolidated bus stop? A consolidated bus stop is a centralized stop that

Consolidated Bus Stops 2019-2020 What is a consolidated bus stop? A consolidated bus stop is a centralized stop that is designed to reduce the ride time to and from school. Sign up and use of these bus stop locations are strictly

354 views • 10 slides

e e e 0 Stop Loss None e e i ni tia l ly , then e e e CFR No ne e e 8 GCM Ma Jo r e

State of South Dakota Employees Benefit Program Ri sk Mi tigation Strategi es Purpose of Study During 2013 legislative sessi on, the Appropriations Committee and Sub Committee on Insurance requested BHR to review stop loss coverage for the

517 views • 25 slides

Suboptimality of Penalized Empirical Risk Minimization in Classification. Guillaume Lecu e

Suboptimality of Penalized Empirical Risk Minimization in Classification. Guillaume Lecu e Universit e Paris 6 COLT 2007, June 13 General Framework. Aggregations Procedures. Optimality in classification. Motivation. M prior estimators

708 views • 54 slides

Consolidated Bus Stops 2015-2016 What is a consolidated bus stop? A consolidated bus stop is a

Consolidated Bus Stops 2015-2016 What is a consolidated bus stop? A consolidated bus stop is a centralized stop that is designed to reduce ride time to and from school. Sign up and use of these bus stop locations are strictly voluntary

116 views • 10 slides

Consolidated Bus Stops 2020-2021 What is a consolidated bus stop? A consolidated bus stop is

Consolidated Bus Stops 2020-2021 What is a consolidated bus stop? A consolidated bus stop is a centralized stop that is designed to reduce the ride time to and from school. Sign up and use of these bus stop locations are strictly

108 views • 10 slides

Group Captive Stop-Loss Gain Control and Transparency While Lowering Your Costs A Presentation

Group Captive Stop-Loss Gain Control and Transparency While Lowering Your Costs A Presentation From CCHI/RMRS 601 Montgomery Street San Francisco, CA 415-398-8985 What is Group Captive Stop-Loss? Its a response to employers who

326 views • 9 slides

Decision Theory, and Loss Functions CMSC 691 UMBC Some slides adapted from Hamed Pirsiavash

Decision Theory, and Loss Functions CMSC 691 UMBC Some slides adapted from Hamed Pirsiavash Todays Goal: learn about empirical risk minimization Set t = 0 Pick a starting value t F Until converged: 1. Get value y t = F( t )

781 views • 49 slides

High Order Methods for Empirical Risk Minimization Alejandro Ribeiro Department of Electrical and

High Order Methods for Empirical Risk Minimization Alejandro Ribeiro Department of Electrical and Systems Engineering University of Pennsylvania aribeiro@seas.upenn.edu Thanks to: Aryan Mokhtari, Mark Eisen, ONR, NSF DIMACS Workshop on

775 views • 40 slides

Loss minimization and parameter estimation with heavy tails Sivan Sabato # Daniel Hsu ? ?

Loss minimization and parameter estimation with heavy tails Sivan Sabato # Daniel Hsu ? ? Department of Computer Science, Columbia University # Microsoft Research New England On the job marketdont miss this amazing hiring opportunity!

568 views • 40 slides

Empirical Method Based Aggregate Loss Distributions C. K. Stan Khury 2012 INTRODUCTION

Empirical Method Based Aggregate Loss Distributions C. K. Stan Khury 2012 INTRODUCTION Aggregate Loss Distributions: Formulaic or Empirical ** Formulaic Distributions: Select Form and Estimate Parameters ** Empirical Distributions:

300 views • 15 slides

1 THE ANALYSIS OF TRAFFIC SAFETY THE ANALYSIS OF TRAFFIC SAFETY 3. Classification of stop

AGENDA AGENDA 1. The role and significance of public transport and tram transport in urban areas 2. Goal and stages of the study 1 THE ANALYSIS OF TRAFFIC SAFETY THE ANALYSIS OF TRAFFIC SAFETY 3. Classification of stop stations at tram stops

169 views • 4 slides

Lecture 24: The Sample Variance S 2 The squared variation 0/ 13 Suppose we have n numbers x 1 , x

Lecture 24: The Sample Variance S 2 The squared variation 0/ 13 Suppose we have n numbers x 1 , x 2 , . . . , x n . Then their squared variation n ( x i x ) 2 sv = sv ( x 1 , x 2 , . . . , x n ) sv ( x 1 , x 2 , . . . , x n ) = i = 1 n

182 views • 14 slides

CSE 401: Introduction to Compiler Construction Course Outline Professor: Craig Chambers

CSE 401: Introduction to Compiler Construction Course Outline Professor: Craig Chambers Front-end issues: lexical analysis (scanning): characters tokens TA: Markus Mock syntax analysis (parsing): tokens abstract syntax trees

1.06k views • 47 slides

RCGC can be more attractive Reference counting example Root set Heap space 1 1 2 1 1 1 1

RCGC can be more attractive Reference counting example Root set Heap space 1 1 2 1 1 1 1 2 1 2 Reference counting example Root set Heap space 1 1 2 1 0 1 1 2 1 3 Reference counting example Root set Heap space 1 1 2 1

620 views • 23 slides

Optimizing Procedure Calls Inlining Procedure calls can be costly (A.k.a. procedure integration,

Optimizing Procedure Calls Inlining Procedure calls can be costly (A.k.a. procedure integration, unfolding, beta-reduction, ...) direct costs of call, return, argument & result passing, stack frame maintainance Replace call with body of

497 views • 4 slides

Matrix Algebra of Sample Statistics James H. Steiger Department of Psychology and Human

Matrix Algebra of Some Sample Statistics Variance of a Linear Combination Variance-Covariance Matrix of Several Linear Combinations Covariance Matrix of Two Sets of Linear Combinations Matrix Algebra of Sample Statistics James H. Steiger

338 views • 29 slides

Variance Will Perkins January 22, 2013 Variance Definition The variance of a random variable X

Variance Will Perkins January 22, 2013 Variance Definition The variance of a random variable X is: var( X ) = E [( X E X ) 2 ] Alternatively, (check using linearity of expectation), var( X ) = E [ X 2 ] ( E X ) 2 Variance Variance is

395 views • 12 slides

Combining Estimates CLRS 2014 Tom Struppeck The University of Texas at Austin Goal: Make a new

Combining Estimates CLRS 2014 Tom Struppeck The University of Texas at Austin Goal: Make a new estimator from old ones Two cases: Case 1: Multiple estimates of one quantity Case 2: Desired quantity is a function of several

367 views • 13 slides

Third quarter 2015 Conference Call Presenters: Yvon Charest, President and CEO Ren Chabot, EVP,

Third quarter 2015 Conference Call Presenters: Yvon Charest, President and CEO Ren Chabot, EVP, CFO and Chief Actuary November 4, 2015 1 Q3 Highlights A solid quarter despite challenging equity markets Expected profit up 14% YoY to $136.4

504 views • 33 slides