LAG: Lazily Aggregated Gradient for Communication-Efficient - - PowerPoint PPT Presentation

lag lazily aggregated gradient for communication
SMART_READER_LITE
LIVE PREVIEW

LAG: Lazily Aggregated Gradient for Communication-Efficient - - PowerPoint PPT Presentation

LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning Tianyi Chen Georgios Giannakis Tao Sun Wotao Yin UMN, ECE UCLA, Math NeurIPS 2018 1 <latexit


slide-1
SLIDE 1

1

Tianyi Chen Georgios Giannakis Tao Sun Wotao Yin UMN, ECE UCLA, Math NeurIPS 2018

LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning

slide-2
SLIDE 2

2

M := {1, . . . , M}

<latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit>

Overview

  • J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al.,

“Large-scale distributed deep networks,” Proc. NIPS., Lake Tahoe, NV, pp. 1223–1231, 2012

slide-3
SLIDE 3

3

M := {1, . . . , M}

<latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit>

Overview

  • J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al.,

“Large-scale distributed deep networks,” Proc. NIPS., Lake Tahoe, NV, pp. 1223–1231, 2012

q Solvers: gradient descent (GD), momentum methods… q Our method improves GD by

§ same convergence rate in theory § reduced communication in theory § more than 90% communication saving in practice

slide-4
SLIDE 4

4

θk

<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>

Vanilla GD implementation

θk

<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>

θk

<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>

θk

<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>

θk+1 = θk α X

m∈M

rLm θk

<latexit sha1_base64="3YCjpjs6cM91HQn3FAIohgUTR/E=">ACYXicbVFNTxsxEPUulI9QYEuPXCwiJKqKaBchwQUpopceWgkAkhxWM06TmLF9q7sWdRotX+SG5de+kfqTXJoISNZfn5vZjx+zgolHcbxaxCurX/Y2Nzabu183N3bjz4d3Lu8tFz0eK5y+5iBE0oa0UOJSjwWVoDOlHjIpt8a/eFZWCdzc4ezQgw0jI0cSQ7oqT6xbJcDd1M+61iOBEI9VM1/ZrUV6uU6SkDVUyAuVKnlWbSVIyDoj/rmhnIFCyOP+pUs0yOT1b2aJQvadSO/E86HuQLEGbLOMmjV7YMOelFga5Auf6SVzgoAKLkitRt1jpRAF8CmPR9CAFm5QzR2q6bFnhnSUW78M0jn7b0UF2jVj+kwNOHFvtYZcpfVLHF0OKmKEoXhi4tGpaKY08ZuOpRWcFQzD4Bb6WelfAIWOPpPaXkTkrdPfg/uzpJ3Eluz9vd6UdW+SQHJETkpAL0iXfyQ3pEU5+B+vBbrAX/Am3wyg8WKSGwbLmM/kvwsO/Z9u58A=</latexit><latexit sha1_base64="3YCjpjs6cM91HQn3FAIohgUTR/E=">ACYXicbVFNTxsxEPUulI9QYEuPXCwiJKqKaBchwQUpopceWgkAkhxWM06TmLF9q7sWdRotX+SG5de+kfqTXJoISNZfn5vZjx+zgolHcbxaxCurX/Y2Nzabu183N3bjz4d3Lu8tFz0eK5y+5iBE0oa0UOJSjwWVoDOlHjIpt8a/eFZWCdzc4ezQgw0jI0cSQ7oqT6xbJcDd1M+61iOBEI9VM1/ZrUV6uU6SkDVUyAuVKnlWbSVIyDoj/rmhnIFCyOP+pUs0yOT1b2aJQvadSO/E86HuQLEGbLOMmjV7YMOelFga5Auf6SVzgoAKLkitRt1jpRAF8CmPR9CAFm5QzR2q6bFnhnSUW78M0jn7b0UF2jVj+kwNOHFvtYZcpfVLHF0OKmKEoXhi4tGpaKY08ZuOpRWcFQzD4Bb6WelfAIWOPpPaXkTkrdPfg/uzpJ3Eluz9vd6UdW+SQHJETkpAL0iXfyQ3pEU5+B+vBbrAX/Am3wyg8WKSGwbLmM/kvwsO/Z9u58A=</latexit><latexit sha1_base64="3YCjpjs6cM91HQn3FAIohgUTR/E=">ACYXicbVFNTxsxEPUulI9QYEuPXCwiJKqKaBchwQUpopceWgkAkhxWM06TmLF9q7sWdRotX+SG5de+kfqTXJoISNZfn5vZjx+zgolHcbxaxCurX/Y2Nzabu183N3bjz4d3Lu8tFz0eK5y+5iBE0oa0UOJSjwWVoDOlHjIpt8a/eFZWCdzc4ezQgw0jI0cSQ7oqT6xbJcDd1M+61iOBEI9VM1/ZrUV6uU6SkDVUyAuVKnlWbSVIyDoj/rmhnIFCyOP+pUs0yOT1b2aJQvadSO/E86HuQLEGbLOMmjV7YMOelFga5Auf6SVzgoAKLkitRt1jpRAF8CmPR9CAFm5QzR2q6bFnhnSUW78M0jn7b0UF2jVj+kwNOHFvtYZcpfVLHF0OKmKEoXhi4tGpaKY08ZuOpRWcFQzD4Bb6WelfAIWOPpPaXkTkrdPfg/uzpJ3Eluz9vd6UdW+SQHJETkpAL0iXfyQ3pEU5+B+vBbrAX/Am3wyg8WKSGwbLmM/kvwsO/Z9u58A=</latexit><latexit sha1_base64="3YCjpjs6cM91HQn3FAIohgUTR/E=">ACYXicbVFNTxsxEPUulI9QYEuPXCwiJKqKaBchwQUpopceWgkAkhxWM06TmLF9q7sWdRotX+SG5de+kfqTXJoISNZfn5vZjx+zgolHcbxaxCurX/Y2Nzabu183N3bjz4d3Lu8tFz0eK5y+5iBE0oa0UOJSjwWVoDOlHjIpt8a/eFZWCdzc4ezQgw0jI0cSQ7oqT6xbJcDd1M+61iOBEI9VM1/ZrUV6uU6SkDVUyAuVKnlWbSVIyDoj/rmhnIFCyOP+pUs0yOT1b2aJQvadSO/E86HuQLEGbLOMmjV7YMOelFga5Auf6SVzgoAKLkitRt1jpRAF8CmPR9CAFm5QzR2q6bFnhnSUW78M0jn7b0UF2jVj+kwNOHFvtYZcpfVLHF0OKmKEoXhi4tGpaKY08ZuOpRWcFQzD4Bb6WelfAIWOPpPaXkTkrdPfg/uzpJ3Eluz9vd6UdW+SQHJETkpAL0iXfyQ3pEU5+B+vBbrAX/Am3wyg8WKSGwbLmM/kvwsO/Z9u58A=</latexit>

rLm

  • θk
<latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="s6cFni5L/+zPwvOelVDQmdiDQvU=">ACDXicbVA9SwNBEJ3z2xg12tosBiE28c5GS8HGwkLBRCEXw9xmkyzZ2zt254Rw5GfY+FdsLBSxtfPfuJekUODZR7vzbAzL0qVtOT7X97C4tLyuraemjvLm1XdkpN2SGS4aPFGJuYvQCiW1aJAkJe5SIzCOlLiNhueFf/sgjJWJvqFRKtox9rXsSY7kpE7lKNQYKcxDjopdjtxGMl+LYwS1bWj2JU8pIEgHN8PC+ewU6n6dX8CNk+CGanCDFedymfYTXgWC01cobWtwE+pnaMhyZUYl8LMihT5EPui5ajGWNh2PjlszA6c0mW9xLiniU3UnxM5xrZY03XGSAP71yvE/7xWRr3Tdi51mpHQfPpRL1OMElakxLrSCE5q5AhyI92ujA/QICeXZcmFEPw9eZ40j+uBXw+ufViDPdiHGgRwAmdwAVfQA6P8Ayv8OY9eS/e+zSuBW+W2y78gvfxDcoNn0c=</latexit><latexit sha1_base64="s6cFni5L/+zPwvOelVDQmdiDQvU=">ACDXicbVA9SwNBEJ3z2xg12tosBiE28c5GS8HGwkLBRCEXw9xmkyzZ2zt254Rw5GfY+FdsLBSxtfPfuJekUODZR7vzbAzL0qVtOT7X97C4tLyuraemjvLm1XdkpN2SGS4aPFGJuYvQCiW1aJAkJe5SIzCOlLiNhueFf/sgjJWJvqFRKtox9rXsSY7kpE7lKNQYKcxDjopdjtxGMl+LYwS1bWj2JU8pIEgHN8PC+ewU6n6dX8CNk+CGanCDFedymfYTXgWC01cobWtwE+pnaMhyZUYl8LMihT5EPui5ajGWNh2PjlszA6c0mW9xLiniU3UnxM5xrZY03XGSAP71yvE/7xWRr3Tdi51mpHQfPpRL1OMElakxLrSCE5q5AhyI92ujA/QICeXZcmFEPw9eZ40j+uBXw+ufViDPdiHGgRwAmdwAVfQA6P8Ayv8OY9eS/e+zSuBW+W2y78gvfxDcoNn0c=</latexit><latexit sha1_base64="AD9+7Z/64N0L6c5CVtCF9t7I5SQ=">ACGHicbVA9TwJBEN3zE/ELtbTZSEywTsbLYk2FhaYyEfCIZlbFtiwt3fZnTMhF36GjX/FxkJjbOn8N+4BhYIv2ezLezOZmRfEUh03W9nZXVtfWMzt5Xf3tnd2y8cHNZNlGjGaySkW4GYLgUitdQoOTNWHMIA8kbwfAm8xtPXBsRqQcxbwdQl+JnmCAVuoUzn0FgYTUZyDp3bgT+oHol/wgkl0zCu2X+jgCOPHYeacdQpFt+xOQZeJNydFMke1U5j43YglIVfIJBjT8twY2yloFEzycd5PDI+BDaHPW5YqCLlp9PDxvTUKl3ai7R9CulU/d2RQmiyNW1lCDgwi14m/ue1EuxdtVOh4gS5YrNBvURSjGiWEu0KzRnKkSXAtLC7UjYADQxtlnkbgrd48jKpX5Q9t+zdu8XK9TyOHDkmJ6REPHJKuSWVEmNMPJMXsk7+XBenDfn0/mala4854j8gfO5AeuvKDG</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit>

rL1

  • θk
<latexit sha1_base64="0izSstp1VXcdyBHas9noR5EY=">ACGHicbVA9SwNBEN2L3/EramzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg=</latexit><latexit sha1_base64="0izSstp1VXcdyBHas9noR5EY=">ACGHicbVA9SwNBEN2L3/EramzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg=</latexit><latexit sha1_base64="0izSstp1VXcdyBHas9noR5EY=">ACGHicbVA9SwNBEN2L3/EramzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg=</latexit><latexit sha1_base64="0izSstp1VXcdyBHas9noR5EY=">ACGHicbVA9SwNBEN2L3/EramzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg=</latexit>

rLM

  • θk
<latexit sha1_base64="s6Z3BoG3QzlpbrNHhZck7zmXId8=">ACGHicbVBNS8NAEN34bf2qevSyWAS91EQEPRa9eFBQsFVoYplst+3SzSbsToQS+jO8+Fe8eFDEa2/+GzdtDlp9sOzjvRlm5oWJFAZd98uZmZ2bX1hcWi6trK6tb5Q3txomTjXjdRbLWN+HYLgUitdRoOT3ieYQhZLfhf3z3L975NqIWN3iIOFBF0lOoIBWqlVPvQVhBIyn4Gkl8PWlR+K7r4fxrJtBpH9Mh97HGH40M+dg1a54lbdMehf4hWkQgpct8ojvx2zNOIKmQRjmp6bYJCBRsEkH5b81PAEWB+6vGmpgoibIBsfNqR7VmnTqztU0jH6s+ODCKTr2krI8CemfZy8T+vmWLnNMiESlLkik0GdVJMaZ5SrQtNGcoB5YA08LuSlkPNDC0WZsCN70yX9J46jquVXv5rhSOyviWCI7ZJfsE4+ckBq5INekTh5Ii/kjbw7z86r8+F8TkpnKJnm/yCM/oGe5ygqg=</latexit><latexit sha1_base64="s6Z3BoG3QzlpbrNHhZck7zmXId8=">ACGHicbVBNS8NAEN34bf2qevSyWAS91EQEPRa9eFBQsFVoYplst+3SzSbsToQS+jO8+Fe8eFDEa2/+GzdtDlp9sOzjvRlm5oWJFAZd98uZmZ2bX1hcWi6trK6tb5Q3txomTjXjdRbLWN+HYLgUitdRoOT3ieYQhZLfhf3z3L975NqIWN3iIOFBF0lOoIBWqlVPvQVhBIyn4Gkl8PWlR+K7r4fxrJtBpH9Mh97HGH40M+dg1a54lbdMehf4hWkQgpct8ojvx2zNOIKmQRjmp6bYJCBRsEkH5b81PAEWB+6vGmpgoibIBsfNqR7VmnTqztU0jH6s+ODCKTr2krI8CemfZy8T+vmWLnNMiESlLkik0GdVJMaZ5SrQtNGcoB5YA08LuSlkPNDC0WZsCN70yX9J46jquVXv5rhSOyviWCI7ZJfsE4+ckBq5INekTh5Ii/kjbw7z86r8+F8TkpnKJnm/yCM/oGe5ygqg=</latexit><latexit sha1_base64="s6Z3BoG3QzlpbrNHhZck7zmXId8=">ACGHicbVBNS8NAEN34bf2qevSyWAS91EQEPRa9eFBQsFVoYplst+3SzSbsToQS+jO8+Fe8eFDEa2/+GzdtDlp9sOzjvRlm5oWJFAZd98uZmZ2bX1hcWi6trK6tb5Q3txomTjXjdRbLWN+HYLgUitdRoOT3ieYQhZLfhf3z3L975NqIWN3iIOFBF0lOoIBWqlVPvQVhBIyn4Gkl8PWlR+K7r4fxrJtBpH9Mh97HGH40M+dg1a54lbdMehf4hWkQgpct8ojvx2zNOIKmQRjmp6bYJCBRsEkH5b81PAEWB+6vGmpgoibIBsfNqR7VmnTqztU0jH6s+ODCKTr2krI8CemfZy8T+vmWLnNMiESlLkik0GdVJMaZ5SrQtNGcoB5YA08LuSlkPNDC0WZsCN70yX9J46jquVXv5rhSOyviWCI7ZJfsE4+ckBq5INekTh5Ii/kjbw7z86r8+F8TkpnKJnm/yCM/oGe5ygqg=</latexit><latexit sha1_base64="s6Z3BoG3QzlpbrNHhZck7zmXId8=">ACGHicbVBNS8NAEN34bf2qevSyWAS91EQEPRa9eFBQsFVoYplst+3SzSbsToQS+jO8+Fe8eFDEa2/+GzdtDlp9sOzjvRlm5oWJFAZd98uZmZ2bX1hcWi6trK6tb5Q3txomTjXjdRbLWN+HYLgUitdRoOT3ieYQhZLfhf3z3L975NqIWN3iIOFBF0lOoIBWqlVPvQVhBIyn4Gkl8PWlR+K7r4fxrJtBpH9Mh97HGH40M+dg1a54lbdMehf4hWkQgpct8ojvx2zNOIKmQRjmp6bYJCBRsEkH5b81PAEWB+6vGmpgoibIBsfNqR7VmnTqztU0jH6s+ODCKTr2krI8CemfZy8T+vmWLnNMiESlLkik0GdVJMaZ5SrQtNGcoB5YA08LuSlkPNDC0WZsCN70yX9J46jquVXv5rhSOyviWCI7ZJfsE4+ckBq5INekTh5Ii/kjbw7z86r8+F8TkpnKJnm/yCM/oGe5ygqg=</latexit>

q Per iteration communication overhead for M uploads (one per worker)

slide-5
SLIDE 5

5

Prior art

Our contribution Adaptively skip communication, provable communication reduction

§ Quantized gradient descent [Kashyap et al., 07], [Alistarh et al., 17], [Suresh et al., 17]… § Increasing computation before communication [Jaggi et al., 14], [Ma et al., 17], [Smith et al., 17]… § Sparse SGD with large entries [Aji-Heafield 17], [Sun et al., 17], [Lin et al., 18], [Stich et al., 18]…

q Communication-efficient distributed learning Ø number of communication rounds is not reduced

slide-6
SLIDE 6

6

Our LAG implementation

θk

<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>

θk

<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>

θk

<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>

rLm

  • θk
<latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="s6cFni5L/+zPwvOelVDQmdiDQvU=">ACDXicbVA9SwNBEJ3z2xg12tosBiE28c5GS8HGwkLBRCEXw9xmkyzZ2zt254Rw5GfY+FdsLBSxtfPfuJekUODZR7vzbAzL0qVtOT7X97C4tLyuraemjvLm1XdkpN2SGS4aPFGJuYvQCiW1aJAkJe5SIzCOlLiNhueFf/sgjJWJvqFRKtox9rXsSY7kpE7lKNQYKcxDjopdjtxGMl+LYwS1bWj2JU8pIEgHN8PC+ewU6n6dX8CNk+CGanCDFedymfYTXgWC01cobWtwE+pnaMhyZUYl8LMihT5EPui5ajGWNh2PjlszA6c0mW9xLiniU3UnxM5xrZY03XGSAP71yvE/7xWRr3Tdi51mpHQfPpRL1OMElakxLrSCE5q5AhyI92ujA/QICeXZcmFEPw9eZ40j+uBXw+ufViDPdiHGgRwAmdwAVfQA6P8Ayv8OY9eS/e+zSuBW+W2y78gvfxDcoNn0c=</latexit><latexit sha1_base64="s6cFni5L/+zPwvOelVDQmdiDQvU=">ACDXicbVA9SwNBEJ3z2xg12tosBiE28c5GS8HGwkLBRCEXw9xmkyzZ2zt254Rw5GfY+FdsLBSxtfPfuJekUODZR7vzbAzL0qVtOT7X97C4tLyuraemjvLm1XdkpN2SGS4aPFGJuYvQCiW1aJAkJe5SIzCOlLiNhueFf/sgjJWJvqFRKtox9rXsSY7kpE7lKNQYKcxDjopdjtxGMl+LYwS1bWj2JU8pIEgHN8PC+ewU6n6dX8CNk+CGanCDFedymfYTXgWC01cobWtwE+pnaMhyZUYl8LMihT5EPui5ajGWNh2PjlszA6c0mW9xLiniU3UnxM5xrZY03XGSAP71yvE/7xWRr3Tdi51mpHQfPpRL1OMElakxLrSCE5q5AhyI92ujA/QICeXZcmFEPw9eZ40j+uBXw+ufViDPdiHGgRwAmdwAVfQA6P8Ayv8OY9eS/e+zSuBW+W2y78gvfxDcoNn0c=</latexit><latexit sha1_base64="AD9+7Z/64N0L6c5CVtCF9t7I5SQ=">ACGHicbVA9TwJBEN3zE/ELtbTZSEywTsbLYk2FhaYyEfCIZlbFtiwt3fZnTMhF36GjX/FxkJjbOn8N+4BhYIv2ezLezOZmRfEUh03W9nZXVtfWMzt5Xf3tnd2y8cHNZNlGjGaySkW4GYLgUitdQoOTNWHMIA8kbwfAm8xtPXBsRqQcxbwdQl+JnmCAVuoUzn0FgYTUZyDp3bgT+oHol/wgkl0zCu2X+jgCOPHYeacdQpFt+xOQZeJNydFMke1U5j43YglIVfIJBjT8twY2yloFEzycd5PDI+BDaHPW5YqCLlp9PDxvTUKl3ai7R9CulU/d2RQmiyNW1lCDgwi14m/ue1EuxdtVOh4gS5YrNBvURSjGiWEu0KzRnKkSXAtLC7UjYADQxtlnkbgrd48jKpX5Q9t+zdu8XK9TyOHDkmJ6REPHJKuSWVEmNMPJMXsk7+XBenDfn0/mala4854j8gfO5AeuvKDG</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit><latexit sha1_base64="I1eQ3hO3bdPUWRLe+PkoBFzGUVo=">ACGHicbVBNS8NAEN34WetX1KOXxSLopSYi6LHoxYOHClaFJpbJdtMu3WzC7kQoT/Di3/FiwdFvPbmv3FTe/DrwbKP92aYmRdlUhj0vA9nZnZufmGxslRdXldW3c3Nq9NmvGWyVqb6NwHApFG+hQMlvM80hiS/iQZnpX9z7URqbrCYcbDBHpKxIBWqnjHgQKIglFwEDSi1EnCSLR2wuiVHbNMLFfEWCfI4zuBqWz3FrXt2bgP4l/pTUyBTNjsOuinLE6QSTCm7XsZhgVoFEzyUTXIDc+ADaDH25YqSLgJi8lhI7prlS6NU2fQjpRv3cUkJhyTVuZAPbNb68U/PaOcYnYSFUliNX7GtQnEuKS1Tol2hOUM5tASYFnZXyvqgaHNsmpD8H+f/JdcH9Z9r+5fHtUap9M4KmSb7JA94pNj0iDnpElahJEH8kReyKvz6Dw7b87V+mM+3ZIj/gjD8Br/ygyg=</latexit>

rL1

  • θk
<latexit sha1_base64="0izSstp1VXcdyBHas9noR5EY=">ACGHicbVA9SwNBEN2L3/EramzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg=</latexit><latexit sha1_base64="0izSstp1VXcdyBHas9noR5EY=">ACGHicbVA9SwNBEN2L3/EramzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg=</latexit><latexit sha1_base64="0izSstp1VXcdyBHas9noR5EY=">ACGHicbVA9SwNBEN2L3/EramzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg=</latexit><latexit sha1_base64="0izSstp1VXcdyBHas9noR5EY=">ACGHicbVA9SwNBEN2L3/EramzGITYxDsRtBRtLCwimA/IxTC32SRL9vaO3TkhHPkZNv4VGwtFbNP5b9xLUmjig2Uf780wMy+IpTDout9Obml5ZXVtfSO/ubW9s1vY26+ZKNGMV1kI90IwHApFK+iQMkbseYQBpLXg8FN5tefuDYiUg84jHkrhJ4SXcEArdQunPoKAgmpz0DSu1Hb8wPRK/lBJDtmGNov9bHPEUaPg8w5aReKbtmdgC4Sb0aKZIZKuzD2OxFLQq6QSTCm6bkxtlLQKJjko7yfGB4DG0CPNy1VEHLTSieHjeixVTq0G2n7FNKJ+rsjhdBka9rKELBv5r1M/M9rJti9bKVCxQlyxaDuomkGNEsJdoRmjOUQ0uAaWF3pawPGhjaLPM2BG/+5EVSOyt7btm7Py9eXc/iWCeH5IiUiEcuyBW5JRVSJYw8k1fyTj6cF+fN+XS+pqU5Z9ZzQP7AGf8ATcigjg=</latexit>

θk+1 = θk α X

m∈Mk

rLm(θk) α X

m∈M/Mk

rLm ˆ θ

k−1 m

  • <latexit sha1_base64="xjayPFLH2d2e1FfhubKeZ30oZTE=">AC2HicbVJdixMxFM2MX2v92KqPvkSL0EW2zMiCvgLvigsILdXWzqcCdNO6FJZkjuCUM+KCIr/403/wV/gUzbcW1OzeEHM65OfmI6+UdJgkv6L4ytVr12/s3ezdun3n7n7/3v1TV9aWizEvVWnPc3BCSPGKFGJ8oK0LkSZ/nyVaufRLWydK8x1UlphoWRs4lBwxU1v/N8lLN3EqHxTMsBELz0S+fps3LmV5yEBVBbBHYbhaZ14zaTzjoOjbIDdrwUCugG7YN02mh51WB/SiWYcdcwpcsWve+1vgnz/L5WLICkDfUSjo4UCHadNmHWT9QTJK1kEvg3QLBmQbJ1n/J5uVvNbCIA/tuEmaVDj1YFyJZoeq52ogC9hISYBGtDCTf36YRr6JDAzOi9tmAbpmr24w4N2bchUwMWbldryS5tUuP8xdRLU9UoDN8UmteKYknbV6YzaQVHtQoAuJWhV8oLsMAx/IVeuIR098iXwemzUZqM0ndHg+Oj7XskYfkMRmSlDwnx+Q1OSFjwqNx5KMv0df4Q/w5/hZ/36TG0XbPA/JfxD/+AHy5xQ=</latexit><latexit sha1_base64="xjayPFLH2d2e1FfhubKeZ30oZTE=">AC2HicbVJdixMxFM2MX2v92KqPvkSL0EW2zMiCvgLvigsILdXWzqcCdNO6FJZkjuCUM+KCIr/403/wV/gUzbcW1OzeEHM65OfmI6+UdJgkv6L4ytVr12/s3ezdun3n7n7/3v1TV9aWizEvVWnPc3BCSPGKFGJ8oK0LkSZ/nyVaufRLWydK8x1UlphoWRs4lBwxU1v/N8lLN3EqHxTMsBELz0S+fps3LmV5yEBVBbBHYbhaZ14zaTzjoOjbIDdrwUCugG7YN02mh51WB/SiWYcdcwpcsWve+1vgnz/L5WLICkDfUSjo4UCHadNmHWT9QTJK1kEvg3QLBmQbJ1n/J5uVvNbCIA/tuEmaVDj1YFyJZoeq52ogC9hISYBGtDCTf36YRr6JDAzOi9tmAbpmr24w4N2bchUwMWbldryS5tUuP8xdRLU9UoDN8UmteKYknbV6YzaQVHtQoAuJWhV8oLsMAx/IVeuIR098iXwemzUZqM0ndHg+Oj7XskYfkMRmSlDwnx+Q1OSFjwqNx5KMv0df4Q/w5/hZ/36TG0XbPA/JfxD/+AHy5xQ=</latexit><latexit sha1_base64="xjayPFLH2d2e1FfhubKeZ30oZTE=">AC2HicbVJdixMxFM2MX2v92KqPvkSL0EW2zMiCvgLvigsILdXWzqcCdNO6FJZkjuCUM+KCIr/403/wV/gUzbcW1OzeEHM65OfmI6+UdJgkv6L4ytVr12/s3ezdun3n7n7/3v1TV9aWizEvVWnPc3BCSPGKFGJ8oK0LkSZ/nyVaufRLWydK8x1UlphoWRs4lBwxU1v/N8lLN3EqHxTMsBELz0S+fps3LmV5yEBVBbBHYbhaZ14zaTzjoOjbIDdrwUCugG7YN02mh51WB/SiWYcdcwpcsWve+1vgnz/L5WLICkDfUSjo4UCHadNmHWT9QTJK1kEvg3QLBmQbJ1n/J5uVvNbCIA/tuEmaVDj1YFyJZoeq52ogC9hISYBGtDCTf36YRr6JDAzOi9tmAbpmr24w4N2bchUwMWbldryS5tUuP8xdRLU9UoDN8UmteKYknbV6YzaQVHtQoAuJWhV8oLsMAx/IVeuIR098iXwemzUZqM0ndHg+Oj7XskYfkMRmSlDwnx+Q1OSFjwqNx5KMv0df4Q/w5/hZ/36TG0XbPA/JfxD/+AHy5xQ=</latexit><latexit sha1_base64="xjayPFLH2d2e1FfhubKeZ30oZTE=">AC2HicbVJdixMxFM2MX2v92KqPvkSL0EW2zMiCvgLvigsILdXWzqcCdNO6FJZkjuCUM+KCIr/403/wV/gUzbcW1OzeEHM65OfmI6+UdJgkv6L4ytVr12/s3ezdun3n7n7/3v1TV9aWizEvVWnPc3BCSPGKFGJ8oK0LkSZ/nyVaufRLWydK8x1UlphoWRs4lBwxU1v/N8lLN3EqHxTMsBELz0S+fps3LmV5yEBVBbBHYbhaZ14zaTzjoOjbIDdrwUCugG7YN02mh51WB/SiWYcdcwpcsWve+1vgnz/L5WLICkDfUSjo4UCHadNmHWT9QTJK1kEvg3QLBmQbJ1n/J5uVvNbCIA/tuEmaVDj1YFyJZoeq52ogC9hISYBGtDCTf36YRr6JDAzOi9tmAbpmr24w4N2bchUwMWbldryS5tUuP8xdRLU9UoDN8UmteKYknbV6YzaQVHtQoAuJWhV8oLsMAx/IVeuIR098iXwemzUZqM0ndHg+Oj7XskYfkMRmSlDwnx+Q1OSFjwqNx5KMv0df4Q/w5/hZ/36TG0XbPA/JfxD/+AHy5xQ=</latexit>

q Select a subset of workers to upload q Remaining workers in do not upload M/Mk

<latexit sha1_base64="qvz+PV4H383Let9dvoj1mypjLYI=">ACAnicbVDLSgMxFL1TX7W+Rl2Jm2ARXJWZUtBlwY0boYJ9QGcsmThmYyQ5IRylDc+CtuXCji1q9w59+YtiNo64HAyTn3uSeIOFMacf5sgorq2vrG8XN0tb2zu6evX/QUnEqCW2SmMeyE2BFORO0qZnmtJNIiqOA03Ywupz67XsqFYvFrR4n1I/wQLCQEayN1LOPMo9gjq4nuJYDdHP9W7Us8tOxZkBLRM3J2XI0ejZn14/JmlEhSZmluq6TqL9DEvNCKeTkpcqmAywgPaNVTgiCo/m60wQadG6aMwluYIjWbq74MR0qNo8BURlgP1aI3Ff/zuqkOL/yMiSTVJD5Q2HKkY7RNA/UZ5ISzceGYCKZ+SsiQywx0Sa1kgnBXVx5mbSqFdepuDe1cr2ax1GEYziBM3DhHOpwBQ1oAoEHeIXeLUerWfrzXqflxasvOcQ/sD6+AaVU5bX</latexit><latexit sha1_base64="qvz+PV4H383Let9dvoj1mypjLYI=">ACAnicbVDLSgMxFL1TX7W+Rl2Jm2ARXJWZUtBlwY0boYJ9QGcsmThmYyQ5IRylDc+CtuXCji1q9w59+YtiNo64HAyTn3uSeIOFMacf5sgorq2vrG8XN0tb2zu6evX/QUnEqCW2SmMeyE2BFORO0qZnmtJNIiqOA03Ywupz67XsqFYvFrR4n1I/wQLCQEayN1LOPMo9gjq4nuJYDdHP9W7Us8tOxZkBLRM3J2XI0ejZn14/JmlEhSZmluq6TqL9DEvNCKeTkpcqmAywgPaNVTgiCo/m60wQadG6aMwluYIjWbq74MR0qNo8BURlgP1aI3Ff/zuqkOL/yMiSTVJD5Q2HKkY7RNA/UZ5ISzceGYCKZ+SsiQywx0Sa1kgnBXVx5mbSqFdepuDe1cr2ax1GEYziBM3DhHOpwBQ1oAoEHeIXeLUerWfrzXqflxasvOcQ/sD6+AaVU5bX</latexit><latexit sha1_base64="qvz+PV4H383Let9dvoj1mypjLYI=">ACAnicbVDLSgMxFL1TX7W+Rl2Jm2ARXJWZUtBlwY0boYJ9QGcsmThmYyQ5IRylDc+CtuXCji1q9w59+YtiNo64HAyTn3uSeIOFMacf5sgorq2vrG8XN0tb2zu6evX/QUnEqCW2SmMeyE2BFORO0qZnmtJNIiqOA03Ywupz67XsqFYvFrR4n1I/wQLCQEayN1LOPMo9gjq4nuJYDdHP9W7Us8tOxZkBLRM3J2XI0ejZn14/JmlEhSZmluq6TqL9DEvNCKeTkpcqmAywgPaNVTgiCo/m60wQadG6aMwluYIjWbq74MR0qNo8BURlgP1aI3Ff/zuqkOL/yMiSTVJD5Q2HKkY7RNA/UZ5ISzceGYCKZ+SsiQywx0Sa1kgnBXVx5mbSqFdepuDe1cr2ax1GEYziBM3DhHOpwBQ1oAoEHeIXeLUerWfrzXqflxasvOcQ/sD6+AaVU5bX</latexit><latexit sha1_base64="qvz+PV4H383Let9dvoj1mypjLYI=">ACAnicbVDLSgMxFL1TX7W+Rl2Jm2ARXJWZUtBlwY0boYJ9QGcsmThmYyQ5IRylDc+CtuXCji1q9w59+YtiNo64HAyTn3uSeIOFMacf5sgorq2vrG8XN0tb2zu6evX/QUnEqCW2SmMeyE2BFORO0qZnmtJNIiqOA03Ywupz67XsqFYvFrR4n1I/wQLCQEayN1LOPMo9gjq4nuJYDdHP9W7Us8tOxZkBLRM3J2XI0ejZn14/JmlEhSZmluq6TqL9DEvNCKeTkpcqmAywgPaNVTgiCo/m60wQadG6aMwluYIjWbq74MR0qNo8BURlgP1aI3Ff/zuqkOL/yMiSTVJD5Q2HKkY7RNA/UZ5ISzceGYCKZ+SsiQywx0Sa1kgnBXVx5mbSqFdepuDe1cr2ax1GEYziBM3DhHOpwBQ1oAoEHeIXeLUerWfrzXqflxasvOcQ/sD6+AaVU5bX</latexit>

Mk ⊆ M

<latexit sha1_base64="KL3ZLMNta0F+BxwcXvA28JXtUA=">ACBXicbVBNS8NAEN3Ur1q/oh71sFgETyUpgh4LXrwIFewHNLFstpN26WYTdzdCb148a948aCIV/+DN/+N2zaCtj4YeLw3w8y8IOFMacf5sgpLyura8X10sbm1vaOvbvXVHEqKTRozGPZDogCzgQ0NMc2okEgUcWsHwYuK37kEqFosbPUrAj0hfsJBRo3UtQ8zjxKOr8a3Q0+lgQINd/hH69plp+JMgReJm5MylHv2p9eL6ZpBEJTpTquE6i/YxIzSiHclLFSEDkfOoYKEoHys+kXY3xslB4OY2lKaDxVf09kJFJqFAWmMyJ6oOa9ifif10l1eO5nTCSpBkFni8KUYx3jS4xyRQzUeGECqZuRXTAZGEahNcyYTgzr+8SJrViutU3OvTcq2ax1FEB+gInSAXnaEaukR1EAUPaAn9IJerUfr2Xqz3metBSuf2Ud/YH18AySFmEY=</latexit><latexit sha1_base64="KL3ZLMNta0F+BxwcXvA28JXtUA=">ACBXicbVBNS8NAEN3Ur1q/oh71sFgETyUpgh4LXrwIFewHNLFstpN26WYTdzdCb148a948aCIV/+DN/+N2zaCtj4YeLw3w8y8IOFMacf5sgpLyura8X10sbm1vaOvbvXVHEqKTRozGPZDogCzgQ0NMc2okEgUcWsHwYuK37kEqFosbPUrAj0hfsJBRo3UtQ8zjxKOr8a3Q0+lgQINd/hH69plp+JMgReJm5MylHv2p9eL6ZpBEJTpTquE6i/YxIzSiHclLFSEDkfOoYKEoHys+kXY3xslB4OY2lKaDxVf09kJFJqFAWmMyJ6oOa9ifif10l1eO5nTCSpBkFni8KUYx3jS4xyRQzUeGECqZuRXTAZGEahNcyYTgzr+8SJrViutU3OvTcq2ax1FEB+gInSAXnaEaukR1EAUPaAn9IJerUfr2Xqz3metBSuf2Ud/YH18AySFmEY=</latexit><latexit sha1_base64="KL3ZLMNta0F+BxwcXvA28JXtUA=">ACBXicbVBNS8NAEN3Ur1q/oh71sFgETyUpgh4LXrwIFewHNLFstpN26WYTdzdCb148a948aCIV/+DN/+N2zaCtj4YeLw3w8y8IOFMacf5sgpLyura8X10sbm1vaOvbvXVHEqKTRozGPZDogCzgQ0NMc2okEgUcWsHwYuK37kEqFosbPUrAj0hfsJBRo3UtQ8zjxKOr8a3Q0+lgQINd/hH69plp+JMgReJm5MylHv2p9eL6ZpBEJTpTquE6i/YxIzSiHclLFSEDkfOoYKEoHys+kXY3xslB4OY2lKaDxVf09kJFJqFAWmMyJ6oOa9ifif10l1eO5nTCSpBkFni8KUYx3jS4xyRQzUeGECqZuRXTAZGEahNcyYTgzr+8SJrViutU3OvTcq2ax1FEB+gInSAXnaEaukR1EAUPaAn9IJerUfr2Xqz3metBSuf2Ud/YH18AySFmEY=</latexit><latexit sha1_base64="KL3ZLMNta0F+BxwcXvA28JXtUA=">ACBXicbVBNS8NAEN3Ur1q/oh71sFgETyUpgh4LXrwIFewHNLFstpN26WYTdzdCb148a948aCIV/+DN/+N2zaCtj4YeLw3w8y8IOFMacf5sgpLyura8X10sbm1vaOvbvXVHEqKTRozGPZDogCzgQ0NMc2okEgUcWsHwYuK37kEqFosbPUrAj0hfsJBRo3UtQ8zjxKOr8a3Q0+lgQINd/hH69plp+JMgReJm5MylHv2p9eL6ZpBEJTpTquE6i/YxIzSiHclLFSEDkfOoYKEoHys+kXY3xslB4OY2lKaDxVf09kJFJqFAWmMyJ6oOa9ifif10l1eO5nTCSpBkFni8KUYx3jS4xyRQzUeGECqZuRXTAZGEahNcyYTgzr+8SJrViutU3OvTcq2ax1FEB+gInSAXnaEaukR1EAUPaAn9IJerUfr2Xqz3metBSuf2Ud/YH18AySFmEY=</latexit>

Old gradient Fresh gradient

slide-7
SLIDE 7

7

LAG: GD under two alternative communication rules

  • T. Chen, G. B. Giannakis, T. Sun, and W. Yin "LAG: Lazily Aggregated Gradient for Communication-Efficient

Distributed Learning," Proc. of NIPS, Montreal, Canada, December 3-8, 2018.

q Worker-side rule (LAG-WK): Include worker m in if

Gradient innovation Optimization progress

  • rLm
  • θk

rLm ˆ θ

k−1 m

  • 1

M

  • 1

α ⇣ θk θk−1⌘

  • <latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="0R4hPLkwHklCrgKU4UNeGnfaJvs=">ACw3icbVJNb9NAEF2brxIKBK5cVlRI7SGRzQWOC5IgBQk0lbKptF4M7ZXWX90d1wpbC3+Izd+AP+DdRwkaD3Sap/em9HMvtmk1spSFP0Kwjt3791/cPBw9Ojw8ZOn42eHp7ZqjMS5rHRlzhOwqFWJc1Kk8bw2CEWi8SzZfOj0sys0VlXlN9rWuCwgK1WqJCnVuPf4r3KxDUXJSQanJCg+ed2VYhEZciqfTabgt/OUE5ErQXm045mQzn50BuoMjrF24zidtdbd9RZHjJRWpAurh1X1qhMSVP/2UE6DqHnh4eZDLE9m2MynI6S9xPRqtxkfRNoFvw3iPThi+5itxj/FupJNgSVJDdYu4qimpQNDSmpsR6KxWIPcQIYLD0so0C7dbh0tf+WZNU8r409JfMf+W+GgsN3YPrMAyu1NrSOHtEVD6dulU2XdEJayb5Q2mlPFu93ytTIoSW89AGmUn5XLHLyh5H9AZ0J8m3wenraRxN468RO2Av2Et2zGL2hr1jH9mMzZkMPgWXwfAhbPwKmx7u8Jg79tz9l+EP/4AYCLkSg=</latexit><latexit sha1_base64="0R4hPLkwHklCrgKU4UNeGnfaJvs=">ACw3icbVJNb9NAEF2brxIKBK5cVlRI7SGRzQWOC5IgBQk0lbKptF4M7ZXWX90d1wpbC3+Izd+AP+DdRwkaD3Sap/em9HMvtmk1spSFP0Kwjt3791/cPBw9Ojw8ZOn42eHp7ZqjMS5rHRlzhOwqFWJc1Kk8bw2CEWi8SzZfOj0sys0VlXlN9rWuCwgK1WqJCnVuPf4r3KxDUXJSQanJCg+ed2VYhEZciqfTabgt/OUE5ErQXm045mQzn50BuoMjrF24zidtdbd9RZHjJRWpAurh1X1qhMSVP/2UE6DqHnh4eZDLE9m2MynI6S9xPRqtxkfRNoFvw3iPThi+5itxj/FupJNgSVJDdYu4qimpQNDSmpsR6KxWIPcQIYLD0so0C7dbh0tf+WZNU8r409JfMf+W+GgsN3YPrMAyu1NrSOHtEVD6dulU2XdEJayb5Q2mlPFu93ytTIoSW89AGmUn5XLHLyh5H9AZ0J8m3wenraRxN468RO2Av2Et2zGL2hr1jH9mMzZkMPgWXwfAhbPwKmx7u8Jg79tz9l+EP/4AYCLkSg=</latexit><latexit sha1_base64="CkJErpo0e1xvgGRANXgvbPzUtg=">ACznicbVJNa9wEJXdr3T7tWmPvYguheSwi51LegzNpdAWtBNAqvNMtbKtlhZdqRxYKuYXv7eusP6P+ovN5Cm3hA6PHePGY0o6RS0mIU/QrCe/cfPHy093jw5Omz5y+G+y/PbFkbLma8VKW5SMAKJbWYoUQlLiojoEiUOE/Wp61+fi2MlaX+iptKLArItEwlB/TUcvibvZcZu6FMQ6LAMQ6KfmqWBUtkdsCSUq3spvCXY5gLhOZy3SqH4/78HND1mLx+6dbjuNl6u4osE1eUpQa4ixv3uWFKpOjpvwDVeXQ0f2NjPvYroyRWY6H3cVuBoPlcBRNom3QuyDegRHZxXQ5/MlWJa8LoZErsHYeRxUuHBiUXIlmwGorKuBryMTcQw2FsAu3XUdD3pmRdPS+KORbtl/HQ4K27btMwvA3N7WrJPm9eYvls4qasaheZdobRWFEva7paupBEc1cYD4Eb6XinPwQ8U/Q9ohxDfvJdcHY0iaNJ/CUanRztxrFHXpM35IDE5JickA9kSmaEBx+Dq+Bb4MJpeB024fcuNQx2nlfkvwh/AFHReVw</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit>

Old gradient

slide-8
SLIDE 8

8

LAG: GD under two alternative communication rules

  • T. Chen, G. B. Giannakis, T. Sun, and W. Yin "LAG: Lazily Aggregated Gradient for Communication-Efficient

Distributed Learning," Proc. of NIPS, Montreal, Canada, December 3-8, 2018.

q Worker-side rule (LAG-WK): q Server-side rule (LAG-PS):

: smoothness of

Include worker m in if § LAG-PS is a sufficient condition for LAG-WK.

Lm

<latexit sha1_base64="IsVjkbiGRCOxdOt70vZudkTBfj4=">AB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJgY2ERwXxgcoS9zSRZsrt37O4J4ci/sLFQxNZ/Y+e/cZNcoYkPBh7vzTAzL0oEN9b3v73C2vrG5lZxu7Szu7d/UD48apo41QwbLBaxbkfUoOAKG5Zbge1EI5WRwFY0vpn5rSfUhsfqwU4SDCUdKj7gjFonPWZdRgW5m/Zkr1zxq/4cZJUEOalAjnqv/NXtxyVqCwT1JhO4Cc2zKi2nAmclrqpwYSyMR1ix1FJZowm18JWdO6ZNBrF0pS+bq74mMSmMmMnKdktqRWfZm4n9eJ7WD6zDjKktKrZYNEgFsTGZvU/6XCOzYuIZq7WwkbU2ZdSGVXAjB8surpHlRDfxqcH9ZqV3mcRThBE7hHAK4ghrcQh0awEDBM7zCm2e8F+/d+1i0Frx85hj+wPv8ATLPkIw=</latexit><latexit sha1_base64="IsVjkbiGRCOxdOt70vZudkTBfj4=">AB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJgY2ERwXxgcoS9zSRZsrt37O4J4ci/sLFQxNZ/Y+e/cZNcoYkPBh7vzTAzL0oEN9b3v73C2vrG5lZxu7Szu7d/UD48apo41QwbLBaxbkfUoOAKG5Zbge1EI5WRwFY0vpn5rSfUhsfqwU4SDCUdKj7gjFonPWZdRgW5m/Zkr1zxq/4cZJUEOalAjnqv/NXtxyVqCwT1JhO4Cc2zKi2nAmclrqpwYSyMR1ix1FJZowm18JWdO6ZNBrF0pS+bq74mMSmMmMnKdktqRWfZm4n9eJ7WD6zDjKktKrZYNEgFsTGZvU/6XCOzYuIZq7WwkbU2ZdSGVXAjB8surpHlRDfxqcH9ZqV3mcRThBE7hHAK4ghrcQh0awEDBM7zCm2e8F+/d+1i0Frx85hj+wPv8ATLPkIw=</latexit><latexit sha1_base64="IsVjkbiGRCOxdOt70vZudkTBfj4=">AB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJgY2ERwXxgcoS9zSRZsrt37O4J4ci/sLFQxNZ/Y+e/cZNcoYkPBh7vzTAzL0oEN9b3v73C2vrG5lZxu7Szu7d/UD48apo41QwbLBaxbkfUoOAKG5Zbge1EI5WRwFY0vpn5rSfUhsfqwU4SDCUdKj7gjFonPWZdRgW5m/Zkr1zxq/4cZJUEOalAjnqv/NXtxyVqCwT1JhO4Cc2zKi2nAmclrqpwYSyMR1ix1FJZowm18JWdO6ZNBrF0pS+bq74mMSmMmMnKdktqRWfZm4n9eJ7WD6zDjKktKrZYNEgFsTGZvU/6XCOzYuIZq7WwkbU2ZdSGVXAjB8surpHlRDfxqcH9ZqV3mcRThBE7hHAK4ghrcQh0awEDBM7zCm2e8F+/d+1i0Frx85hj+wPv8ATLPkIw=</latexit><latexit sha1_base64="IsVjkbiGRCOxdOt70vZudkTBfj4=">AB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJgY2ERwXxgcoS9zSRZsrt37O4J4ci/sLFQxNZ/Y+e/cZNcoYkPBh7vzTAzL0oEN9b3v73C2vrG5lZxu7Szu7d/UD48apo41QwbLBaxbkfUoOAKG5Zbge1EI5WRwFY0vpn5rSfUhsfqwU4SDCUdKj7gjFonPWZdRgW5m/Zkr1zxq/4cZJUEOalAjnqv/NXtxyVqCwT1JhO4Cc2zKi2nAmclrqpwYSyMR1ix1FJZowm18JWdO6ZNBrF0pS+bq74mMSmMmMnKdktqRWfZm4n9eJ7WD6zDjKktKrZYNEgFsTGZvU/6XCOzYuIZq7WwkbU2ZdSGVXAjB8surpHlRDfxqcH9ZqV3mcRThBE7hHAK4ghrcQh0awEDBM7zCm2e8F+/d+1i0Frx85hj+wPv8ATLPkIw=</latexit>

Include worker m in if

Gradient innovation Optimization progress

  • rLm
  • θk

rLm ˆ θ

k−1 m

  • 1

M

  • 1

α ⇣ θk θk−1⌘

  • <latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="0R4hPLkwHklCrgKU4UNeGnfaJvs=">ACw3icbVJNb9NAEF2brxIKBK5cVlRI7SGRzQWOC5IgBQk0lbKptF4M7ZXWX90d1wpbC3+Izd+AP+DdRwkaD3Sap/em9HMvtmk1spSFP0Kwjt3791/cPBw9Ojw8ZOn42eHp7ZqjMS5rHRlzhOwqFWJc1Kk8bw2CEWi8SzZfOj0sys0VlXlN9rWuCwgK1WqJCnVuPf4r3KxDUXJSQanJCg+ed2VYhEZciqfTabgt/OUE5ErQXm045mQzn50BuoMjrF24zidtdbd9RZHjJRWpAurh1X1qhMSVP/2UE6DqHnh4eZDLE9m2MynI6S9xPRqtxkfRNoFvw3iPThi+5itxj/FupJNgSVJDdYu4qimpQNDSmpsR6KxWIPcQIYLD0so0C7dbh0tf+WZNU8r409JfMf+W+GgsN3YPrMAyu1NrSOHtEVD6dulU2XdEJayb5Q2mlPFu93ytTIoSW89AGmUn5XLHLyh5H9AZ0J8m3wenraRxN468RO2Av2Et2zGL2hr1jH9mMzZkMPgWXwfAhbPwKmx7u8Jg79tz9l+EP/4AYCLkSg=</latexit><latexit sha1_base64="0R4hPLkwHklCrgKU4UNeGnfaJvs=">ACw3icbVJNb9NAEF2brxIKBK5cVlRI7SGRzQWOC5IgBQk0lbKptF4M7ZXWX90d1wpbC3+Izd+AP+DdRwkaD3Sap/em9HMvtmk1spSFP0Kwjt3791/cPBw9Ojw8ZOn42eHp7ZqjMS5rHRlzhOwqFWJc1Kk8bw2CEWi8SzZfOj0sys0VlXlN9rWuCwgK1WqJCnVuPf4r3KxDUXJSQanJCg+ed2VYhEZciqfTabgt/OUE5ErQXm045mQzn50BuoMjrF24zidtdbd9RZHjJRWpAurh1X1qhMSVP/2UE6DqHnh4eZDLE9m2MynI6S9xPRqtxkfRNoFvw3iPThi+5itxj/FupJNgSVJDdYu4qimpQNDSmpsR6KxWIPcQIYLD0so0C7dbh0tf+WZNU8r409JfMf+W+GgsN3YPrMAyu1NrSOHtEVD6dulU2XdEJayb5Q2mlPFu93ytTIoSW89AGmUn5XLHLyh5H9AZ0J8m3wenraRxN468RO2Av2Et2zGL2hr1jH9mMzZkMPgWXwfAhbPwKmx7u8Jg79tz9l+EP/4AYCLkSg=</latexit><latexit sha1_base64="CkJErpo0e1xvgGRANXgvbPzUtg=">ACznicbVJNa9wEJXdr3T7tWmPvYguheSwi51LegzNpdAWtBNAqvNMtbKtlhZdqRxYKuYXv7eusP6P+ovN5Cm3hA6PHePGY0o6RS0mIU/QrCe/cfPHy093jw5Omz5y+G+y/PbFkbLma8VKW5SMAKJbWYoUQlLiojoEiUOE/Wp61+fi2MlaX+iptKLArItEwlB/TUcvibvZcZu6FMQ6LAMQ6KfmqWBUtkdsCSUq3spvCXY5gLhOZy3SqH4/78HND1mLx+6dbjuNl6u4osE1eUpQa4ixv3uWFKpOjpvwDVeXQ0f2NjPvYroyRWY6H3cVuBoPlcBRNom3QuyDegRHZxXQ5/MlWJa8LoZErsHYeRxUuHBiUXIlmwGorKuBryMTcQw2FsAu3XUdD3pmRdPS+KORbtl/HQ4K27btMwvA3N7WrJPm9eYvls4qasaheZdobRWFEva7paupBEc1cYD4Eb6XinPwQ8U/Q9ohxDfvJdcHY0iaNJ/CUanRztxrFHXpM35IDE5JickA9kSmaEBx+Dq+Bb4MJpeB024fcuNQx2nlfkvwh/AFHReVw</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit>

Lm

  • θk − ˆ

θ

k−1 m

  • ≥ 1

M

  • 1

α ⇣ θk − θk−1⌘

  • <latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="EXy2rVIBegRSOrMj8+MSFv1Xw2g=">ACknicdZHPbtQwEMad8K8sBRaOcMCiQiqHrpJe4FiBkAVCS2rbTeRhPvJLHWToI9QVqleSuehBtvg7NZJGiXkSx/+o1HM/4mrbVyFEW/gvDGzVu37+zcHd3bvf/g4fjR7qmrGitxKitd2fMUHGpV4pQUaTyvLYJNZ6ly7d9/uw7Wqeq8iutapwbyEuVKQnkUTL+cZwY8Ubl4pKLtNILtzL+agUVSNBdLA9EAdRuSXWJuWiXB3E3VIscv3GRWZBt3LWfOqExI4/EAG6LmDA+/ptIUOHazKC3o5XOJyNErGe9EkWge/LuKN2GObOEnGP8Wiko3BkqQG52ZxVNO8BUtKauxGonFYg1xCjMvSzDo5u3a3Y6/8GTBs8r6UxJf078rWjCuH9u/NECFu5r4bcrKHs9bxVZd0QlnJolDWaU8X7VfGFsihJr7wAaZWflcsCvKHkF9qbEF/98nVxejiJo0n8JWI7Cl7zvZzF6xI/aenbApk8GT4Cj4EHwMn4XvwuPBrjDY+PaY/RPh598qd9Cj</latexit><latexit sha1_base64="EXy2rVIBegRSOrMj8+MSFv1Xw2g=">ACknicdZHPbtQwEMad8K8sBRaOcMCiQiqHrpJe4FiBkAVCS2rbTeRhPvJLHWToI9QVqleSuehBtvg7NZJGiXkSx/+o1HM/4mrbVyFEW/gvDGzVu37+zcHd3bvf/g4fjR7qmrGitxKitd2fMUHGpV4pQUaTyvLYJNZ6ly7d9/uw7Wqeq8iutapwbyEuVKQnkUTL+cZwY8Ubl4pKLtNILtzL+agUVSNBdLA9EAdRuSXWJuWiXB3E3VIscv3GRWZBt3LWfOqExI4/EAG6LmDA+/ptIUOHazKC3o5XOJyNErGe9EkWge/LuKN2GObOEnGP8Wiko3BkqQG52ZxVNO8BUtKauxGonFYg1xCjMvSzDo5u3a3Y6/8GTBs8r6UxJf078rWjCuH9u/NECFu5r4bcrKHs9bxVZd0QlnJolDWaU8X7VfGFsihJr7wAaZWflcsCvKHkF9qbEF/98nVxejiJo0n8JWI7Cl7zvZzF6xI/aenbApk8GT4Cj4EHwMn4XvwuPBrjDY+PaY/RPh598qd9Cj</latexit><latexit sha1_base64="FwqDNGX8HsaSJO47Ghwur+5XsMo=">ACnXicdVFNj9MwEHXC1K+unCDAxYV0nLYKtkLHCsQEqAFLRLdrlR3o4nrJFbtJGtPkCpv/hW/hBv/BqcpEuyWkax5em9GM36T1kpajKJfQXj5q3bd/buDu7df/Dw0XD/8amtGsPFlFeqMmcpWKFkKaYoUYmz2gjQqRKzdPWu02fhbGyKr/huhYLDXkpM8kBPZUMfxwnmr2VObukLK3U0q61T45hIRDa89UhKwDdDqlN9LlbHcZt381ycUFZoC7uHWfW6ZEhp7+wzBQdQE9fCfSTvYfoKReYGv+sQuB4NkOIrG0SbodRBvwYhs4yQZ/mTLijdalMgVWDuPoxoXDgxKrkQ7YI0VNfAV5GLuYQla2IXbuNvSl5Z0qwy/pVIN+zfHQ607db2lRqwsFe1jtylzRvM3iycLOsGRcn7QVmjKFa0OxVdSiM4qrUHwI30u1JegDcU/UE7E+KrX74OTo/GcTSOv0ajydHWj3yjLwgByQmr8mEfCAnZEp48DSYB+DT+Hz8H14H7pS8Ng2/OE/BPh7DdAEdGY</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit>

Old gradient

slide-9
SLIDE 9

9

Iteration and communication complexity

(nonconvex) Local loss is smooth. (convex) Loss is convex. (strongly convex) Loss is (restricted) strongly convex. In all cases, LAG enjoys the same convergence rate as GD.

Theorem 1

If local objectives are heterogeneous, LAG requires smaller number of uploads to a given accuracy than GD; e.g., as small as 1/M.

Theorem 2

slide-10
SLIDE 10

10

Linear prediction

q Real datasets distributed on M = 9 workers

  • M. Lichman, “UCI machine learning repository,” 2013. [Online]: http://archive.ics.uci.edu/ml

Cyc-/Num-IAG: cyclic/non-uniform update of incremental aggregated gradient

slide-11
SLIDE 11

11

Logistic regression

Ø LAG needs same number of iterations but fewer uploads q Real datasets distributed on M = 9 workers

slide-12
SLIDE 12

12

Conclusions

q Adaptive communication rules for distributed learning q Not degrade convergence but reduce communication

Th Thank You!

Thu Dec 6th 05:00 -- 07:00 PM @ Room 210 & 230 AB #8