lag lazily aggregated gradient for communication
play

LAG: Lazily Aggregated Gradient for Communication-Efficient - PowerPoint PPT Presentation

Nov 19, 2023 •115 likes •246 views

LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning Tianyi Chen Georgios Giannakis Tao Sun Wotao Yin UMN, ECE UCLA, Math NeurIPS 2018 1 <latexit

  1. LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning Tianyi Chen Georgios Giannakis Tao Sun Wotao Yin UMN, ECE UCLA, Math NeurIPS 2018 1

  2. <latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit> <latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit> <latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit> <latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit> Overview M := { 1 , . . . , M } J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al., 2 “Large-scale distributed deep networks,” Proc. NIPS ., Lake Tahoe, NV, pp. 1223–1231, 2012

  3. <latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit> <latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit> <latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit> <latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit> Overview M := { 1 , . . . , M } q Solvers: gradient descent (GD), momentum methods… q Our method improves GD by § same convergence rate in theory § reduced communication in theory § more than 90% communication saving in practice J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al., 3 “Large-scale distributed deep networks,” Proc. NIPS ., Lake Tahoe, NV, pp. 1223–1231, 2012

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Frame- -Aggregated Concurrent Aggregated Concurrent Frame Matching Switch Matching Switch Bill

Frame- -Aggregated Concurrent Aggregated Concurrent Frame Matching Switch Matching Switch Bill

Frame- -Aggregated Concurrent Aggregated Concurrent Frame Matching Switch Matching Switch Bill Lin (University of California, San Diego) Isaac Keslassy (Technion, Israel) Background Background The Concurrent Matching Switch (CMS)

493 views • 38 slides
SK Telecom 1 U U U U U U U- U - - communication - - - - - communication

SK Telecom 1 U U U U U U U- U - - communication - - - - - communication

U U U U U U U- U - - communication - - - - - communication communication communication communication communication communication communication Korea Mobile Market Trend for Convergence &amp; Ubiquitous Communication 2004.

400 views • 16 slides
Data structure Mapping data What data you need to entry Exact location (case specific)

Data structure Mapping data What data you need to entry Exact location (case specific)

Attributes and features Data structure Mapping data What data you need to entry Exact location (case specific) Aggregated data by administrative levels (aggregated) Aggregated data are usually represented by polygons A field for the

673 views • 44 slides
Gradient Analysis NMDS Indirect Gradient Analysis NMDS Direct Gradient Analysis Objective:

Gradient Analysis NMDS Indirect Gradient Analysis NMDS Direct Gradient Analysis Objective:

Multivariate Fundamentals: Rotation/Distance Gradient Analysis NMDS Indirect Gradient Analysis NMDS Direct Gradient Analysis Objective: Use one dataset to explain another Use the spatial patterns of each dataset to try and understand the

364 views • 24 slides
Analyzing Aggregated AR(1) Processes Jon Gunnip Supervisory Committee Professor Lajos Horv

Analyzing Aggregated AR(1) Processes Jon Gunnip Supervisory Committee Professor Lajos Horv

Analyzing Aggregated AR(1) Processes Jon Gunnip Supervisory Committee Professor Lajos Horv ath (Committee Chair) Professor Davar Koshnevisan Professor Paul Roberts Analyzing Aggregated AR(1) Processes p.1 What is an AR(1) Process?

682 views • 57 slides
Weakening Aggregated Traffic of Weakening Aggregated Traffic of DHCP Discover Messages draft

Weakening Aggregated Traffic of Weakening Aggregated Traffic of DHCP Discover Messages draft

Weakening Aggregated Traffic of Weakening Aggregated Traffic of DHCP Discover Messages draft yang sunset4 weaken dhcp 00 draft yang sunset4 weaken dhcp 00 Tianle Yang, Lianyuan Li, Qiongfang Ma China Mobile China Mobile

535 views • 12 slides
How to use Gradient and Multi-Texture 1. Many situations, we need use the gradient texture for our

How to use Gradient and Multi-Texture 1. Many situations, we need use the gradient texture for our

How to use Gradient and Multi-Texture 1. Many situations, we need use the gradient texture for our shape or text, so let s learn how to use our gradient and multi-texture functions. First, crate a shape. 2. Select this shape, switch to color

196 views • 3 slides
Communication-Efficient Decentralized Learning Yuejie Chi EdgeComm Workshop, 2020

Communication-Efficient Decentralized Learning Yuejie Chi EdgeComm Workshop, 2020

Communication-Efficient Decentralized Learning Yuejie Chi EdgeComm Workshop, 2020 Acknowledgements Boyue Li Shicong Cen Yuxin Chen CMU CMU Princeton Communication-Efficient Distributed Optimization in Networks with Gradient Tracking and

627 views • 29 slides
Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap

Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap

CS6501: Deep Learning for Visual Recognition Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap Regression vs Classification Generalization / Overfitting / Underfitting Regularization

633 views • 30 slides
Conjugate Gradient (CG) Majid Lesani Alireza Masoum Overview Backpropagation Gradient

Conjugate Gradient (CG) Majid Lesani Alireza Masoum Overview Backpropagation Gradient

Conjugate Gradient (CG) Majid Lesani Alireza Masoum Overview Backpropagation Gradient Descent Quadratic Forms Gradient Descent in Quadratic Forms Eigen vectors and values Gradient Descent Convergence Conjugate

1.1k views • 50 slides
Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to

Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to

Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to Data Science Pavlos Protopapas, Kevin Rader and Chris Tanner New requirement for the final project: For the first time ever, researchers who submit

619 views • 34 slides
1 Gradient descent with fixed step In this section, we discuss a gradient descent method with

1 Gradient descent with fixed step In this section, we discuss a gradient descent method with

MAP562 Optimal design of structures by Gr egoire Allaire, Thomas Wick Ecole Polytechnique Academic year 2016-2017 Gradient descent and Newtons method for minimization in a PDE / function space context We recapitulate the basic steps to

327 views • 5 slides
Scalability analysis of the distributed-memory implementation of the Aggregated unfitted Finite

Scalability analysis of the distributed-memory implementation of the Aggregated unfitted Finite

Scalability analysis of the distributed-memory implementation of the Aggregated unfitted Finite Element Method (AgFEM) Alberto F. Martn , Santiago Badia, Francesc Verdugo, Eric Neiva MWNDEA 2020, Melbourne, Australia, 12/02/2020 Embedded

899 views • 51 slides
APCNN : Tackling Class Imbalance in Relation Extraction through Aggregated Piecewise Convolutional

APCNN : Tackling Class Imbalance in Relation Extraction through Aggregated Piecewise Convolutional

APCNN : Tackling Class Imbalance in Relation Extraction through Aggregated Piecewise Convolutional Neural Networks Alisa Smirnova , Julien Audiffren, Philippe Cudr-Mauroux eXascale Infolab, University of Fribourg, Switzerland Table of Contents

389 views • 27 slides
OSMOSIS and DIFFUSION Concentration gradient Concentration Gradient - change in the concentration

OSMOSIS and DIFFUSION Concentration gradient Concentration Gradient - change in the concentration

OSMOSIS and DIFFUSION Concentration gradient Concentration Gradient - change in the concentration of a substance from one area to another. Molecules are always moving Molecules move randomly and bump into each other and other barriers

135 views • 13 slides
Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade

Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade

Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade 2018 c University of Washington cse446-staff@cs.washington.edu 1 / 12 Announcements Midterm: Weds, Feb 7th. Policies: You may use a single

333 views • 18 slides
SeparatingThickness fromGeometricThickness DavidEppstein

SeparatingThickness fromGeometricThickness DavidEppstein

Separatingthicknessfromgeometricthickness D.Eppstein,UCIrvine,GraphDrawing2002 SeparatingThickness fromGeometricThickness DavidEppstein Univ.ofCalifornia,Irvine

489 views • 20 slides
compsci 514: algorithms for data science Cameron Musco University of Massachusetts Amherst.

compsci 514: algorithms for data science Cameron Musco University of Massachusetts Amherst.

compsci 514: algorithms for data science Cameron Musco University of Massachusetts Amherst. Spring 2020. Lecture 23 0 summary Last Class: convexity and Lipschitzness. This Class: optimization. 1 Multivariable calculus review and

271 views • 13 slides
CS7015 (Deep Learning) : Lecture 5 Gradient Descent (GD), Momentum Based GD, Nesterov Accelerated

CS7015 (Deep Learning) : Lecture 5 Gradient Descent (GD), Momentum Based GD, Nesterov Accelerated

CS7015 (Deep Learning) : Lecture 5 Gradient Descent (GD), Momentum Based GD, Nesterov Accelerated GD, Stochastic GD, AdaGrad, RMSProp, Adam Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute of Technology Madras 1

1.24k views • 94 slides
Amir Ali Kouzeh Geran and Arash Reyhani-Masoleh Presented by: Arash Reyhani-Masoleh Department

Amir Ali Kouzeh Geran and Arash Reyhani-Masoleh Presented by: Arash Reyhani-Masoleh Department

Amir Ali Kouzeh Geran and Arash Reyhani-Masoleh Presented by: Arash Reyhani-Masoleh Department of Electrical and Computer Engineering Western University, London, Ontario, Canada 23rd IEEE Symposium on Computer Arithmetic (ARITH 23) June 11,

240 views • 20 slides
PA-GD: On the Convergence of Perturbed Alternating Gradient Descent to Second-Order Stationary

PA-GD: On the Convergence of Perturbed Alternating Gradient Descent to Second-Order Stationary

ICML 2019 PA-GD: On the Convergence of Perturbed Alternating Gradient Descent to Second-Order Stationary Points for Structured Nonconvex Optimization Presenter: Songtao Lu University of Minnesota Twin Cities Joint work with Mingyi Hong and

1.05k views • 39 slides
The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima

The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima

The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects Zhanxing Zhu , Jingfeng Wu , Bing Yu, Lei Wu, Jinwen Ma. Peking University Beijing Institute of Big Data Research

319 views • 15 slides
Walk alkin ing Ran andomly ly, Mas Massiv ively ly, an and Effic iciently ly Jakub

Walk alkin ing Ran andomly ly, Mas Massiv ively ly, an and Effic iciently ly Jakub

Walk alkin ing Ran andomly ly, Mas Massiv ively ly, an and Effic iciently ly Jakub cki Slobodan Mitrovi Krzysztof Onak Piotr Sankowski Why Random Walks? Web ratings [Page, Brin, Motwani, Winograd 99] [Berkhin 05]

888 views • 49 slides
materials for magnetic refrigeration Ekkes Brck Introduction Magnetic cooling Giant

materials for magnetic refrigeration Ekkes Brck Introduction Magnetic cooling Giant

Magnetocaloric effect materials for magnetic refrigeration Ekkes Brck Introduction Magnetic cooling Giant magnetocaloric effect Outlook Review: E. Brck, Magnetic refrigeration near room temperature, Handbook of magnetic materials Vol 17

1.05k views • 57 slides

More recommend