LAG: Lazily Aggregated Gradient for Communication-Efficient - - PowerPoint PPT Presentation
LAG: Lazily Aggregated Gradient for Communication-Efficient - - PowerPoint PPT Presentation
LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning Tianyi Chen Georgios Giannakis Tao Sun Wotao Yin UMN, ECE UCLA, Math NeurIPS 2018 1 <latexit
2
M := {1, . . . , M}
<latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit>Overview
- J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al.,
“Large-scale distributed deep networks,” Proc. NIPS., Lake Tahoe, NV, pp. 1223–1231, 2012
3
M := {1, . . . , M}
<latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit><latexit sha1_base64="K7JrU5gtA9yB+iPfJpt/ZlnO0yY=">ACAXicbVDLSsNAFJ3UV62vqBvBzWARXJSERhIbN4UK9gFNKJPJpB06mQkzE6GEuvFX3LhQxK1/4c6/cdpmoa0HLhzOuZd7wkSRpV2nG+rsLK6tr5R3Cxtbe/s7tn7B20lUolJCwsmZDdAijDKSUtTzUg3kQTFASOdYHQz9TsPRCoq+L0eJ8SP0YDTiGKkjdS3jzIPIwYbk6trL3MrHguFVpWGN+nbZafqzACXiZuTMsjR7NtfXihwGhOuMUNK9Vwn0X6GpKaYkUnJSxVJEB6hAekZylFMlJ/NPpjAU6OEMBLSFNdwpv6eyFCs1DgOTGeM9FAtelPxP6+X6ujSzyhPUk04ni+KUga1gNM4YEglwZqNDUFYUnMrxEMkEdYmtJIJwV18eZm0z6uU3XvauV6LY+jCI7BCTgDLrgAdXALmqAFMHgEz+AVvFlP1ov1bn3MWwtWPnMI/sD6/AEieJX9</latexit>Overview
- J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al.,
“Large-scale distributed deep networks,” Proc. NIPS., Lake Tahoe, NV, pp. 1223–1231, 2012
q Solvers: gradient descent (GD), momentum methods… q Our method improves GD by
§ same convergence rate in theory § reduced communication in theory § more than 90% communication saving in practice
4
θk
<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>Vanilla GD implementation
θk
<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>θk
<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>θk
<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>θk+1 = θk α X
m∈M
rLm θk
<latexit sha1_base64="3YCjpjs6cM91HQn3FAIohgUTR/E=">ACYXicbVFNTxsxEPUulI9QYEuPXCwiJKqKaBchwQUpopceWgkAkhxWM06TmLF9q7sWdRotX+SG5de+kfqTXJoISNZfn5vZjx+zgolHcbxaxCurX/Y2Nzabu183N3bjz4d3Lu8tFz0eK5y+5iBE0oa0UOJSjwWVoDOlHjIpt8a/eFZWCdzc4ezQgw0jI0cSQ7oqT6xbJcDd1M+61iOBEI9VM1/ZrUV6uU6SkDVUyAuVKnlWbSVIyDoj/rmhnIFCyOP+pUs0yOT1b2aJQvadSO/E86HuQLEGbLOMmjV7YMOelFga5Auf6SVzgoAKLkitRt1jpRAF8CmPR9CAFm5QzR2q6bFnhnSUW78M0jn7b0UF2jVj+kwNOHFvtYZcpfVLHF0OKmKEoXhi4tGpaKY08ZuOpRWcFQzD4Bb6WelfAIWOPpPaXkTkrdPfg/uzpJ3Eluz9vd6UdW+SQHJETkpAL0iXfyQ3pEU5+B+vBbrAX/Am3wyg8WKSGwbLmM/kvwsO/Z9u58A=</latexit><latexit sha1_base64="3YCjpjs6cM91HQn3FAIohgUTR/E=">ACYXicbVFNTxsxEPUulI9QYEuPXCwiJKqKaBchwQUpopceWgkAkhxWM06TmLF9q7sWdRotX+SG5de+kfqTXJoISNZfn5vZjx+zgolHcbxaxCurX/Y2Nzabu183N3bjz4d3Lu8tFz0eK5y+5iBE0oa0UOJSjwWVoDOlHjIpt8a/eFZWCdzc4ezQgw0jI0cSQ7oqT6xbJcDd1M+61iOBEI9VM1/ZrUV6uU6SkDVUyAuVKnlWbSVIyDoj/rmhnIFCyOP+pUs0yOT1b2aJQvadSO/E86HuQLEGbLOMmjV7YMOelFga5Auf6SVzgoAKLkitRt1jpRAF8CmPR9CAFm5QzR2q6bFnhnSUW78M0jn7b0UF2jVj+kwNOHFvtYZcpfVLHF0OKmKEoXhi4tGpaKY08ZuOpRWcFQzD4Bb6WelfAIWOPpPaXkTkrdPfg/uzpJ3Eluz9vd6UdW+SQHJETkpAL0iXfyQ3pEU5+B+vBbrAX/Am3wyg8WKSGwbLmM/kvwsO/Z9u58A=</latexit><latexit sha1_base64="3YCjpjs6cM91HQn3FAIohgUTR/E=">ACYXicbVFNTxsxEPUulI9QYEuPXCwiJKqKaBchwQUpopceWgkAkhxWM06TmLF9q7sWdRotX+SG5de+kfqTXJoISNZfn5vZjx+zgolHcbxaxCurX/Y2Nzabu183N3bjz4d3Lu8tFz0eK5y+5iBE0oa0UOJSjwWVoDOlHjIpt8a/eFZWCdzc4ezQgw0jI0cSQ7oqT6xbJcDd1M+61iOBEI9VM1/ZrUV6uU6SkDVUyAuVKnlWbSVIyDoj/rmhnIFCyOP+pUs0yOT1b2aJQvadSO/E86HuQLEGbLOMmjV7YMOelFga5Auf6SVzgoAKLkitRt1jpRAF8CmPR9CAFm5QzR2q6bFnhnSUW78M0jn7b0UF2jVj+kwNOHFvtYZcpfVLHF0OKmKEoXhi4tGpaKY08ZuOpRWcFQzD4Bb6WelfAIWOPpPaXkTkrdPfg/uzpJ3Eluz9vd6UdW+SQHJETkpAL0iXfyQ3pEU5+B+vBbrAX/Am3wyg8WKSGwbLmM/kvwsO/Z9u58A=</latexit><latexit sha1_base64="3YCjpjs6cM91HQn3FAIohgUTR/E=">ACYXicbVFNTxsxEPUulI9QYEuPXCwiJKqKaBchwQUpopceWgkAkhxWM06TmLF9q7sWdRotX+SG5de+kfqTXJoISNZfn5vZjx+zgolHcbxaxCurX/Y2Nzabu183N3bjz4d3Lu8tFz0eK5y+5iBE0oa0UOJSjwWVoDOlHjIpt8a/eFZWCdzc4ezQgw0jI0cSQ7oqT6xbJcDd1M+61iOBEI9VM1/ZrUV6uU6SkDVUyAuVKnlWbSVIyDoj/rmhnIFCyOP+pUs0yOT1b2aJQvadSO/E86HuQLEGbLOMmjV7YMOelFga5Auf6SVzgoAKLkitRt1jpRAF8CmPR9CAFm5QzR2q6bFnhnSUW78M0jn7b0UF2jVj+kwNOHFvtYZcpfVLHF0OKmKEoXhi4tGpaKY08ZuOpRWcFQzD4Bb6WelfAIWOPpPaXkTkrdPfg/uzpJ3Eluz9vd6UdW+SQHJETkpAL0iXfyQ3pEU5+B+vBbrAX/Am3wyg8WKSGwbLmM/kvwsO/Z9u58A=</latexit>rLm
- θk
rL1
- θk
rLM
- θk
q Per iteration communication overhead for M uploads (one per worker)
5
Prior art
Our contribution Adaptively skip communication, provable communication reduction
§ Quantized gradient descent [Kashyap et al., 07], [Alistarh et al., 17], [Suresh et al., 17]… § Increasing computation before communication [Jaggi et al., 14], [Ma et al., 17], [Smith et al., 17]… § Sparse SGD with large entries [Aji-Heafield 17], [Sun et al., 17], [Lin et al., 18], [Stich et al., 18]…
q Communication-efficient distributed learning Ø number of communication rounds is not reduced
6
Our LAG implementation
θk
<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>θk
<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>θk
<latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit><latexit sha1_base64="QFufTeJ1sA/bV2evRx5P9WHmcSE=">AB/nicbVDLSsNAFJ3UV62vqLhyM1gEVyURQZdFNy4r2Ac0sUwmk3boZBJmboQSCv6KGxeKuPU73Pk3TtostPXAMIdz7mXOnCAVXIPjfFuVldW19Y3qZm1re2d3z94/6OgkU5S1aSIS1QuIZoJL1gYOgvVSxUgcCNYNxjeF31kSvNE3sMkZX5MhpJHnBIw0sA+8oJEhHoSmyv3YMSATB/GA7vuNJwZ8DJxS1JHJVoD+8sLE5rFTAIVROu+6Tg50QBp4JNa16mWUromAxZ31BJYqb9fBZ/ik+NEuIoUeZIwDP190ZOYl0kNJMxgZFe9ArxP6+fQXTl51ymGTBJ5w9FmcCQ4KILHLFKIiJIYQqbrJiOiKUDCN1UwJ7uKXl0nvOE6Dfuot68LuomN0gs6Qiy5RE92iFmojinL0jF7Rm/VkvVjv1sd8tGKVO4foD6zPHzBnlkM=</latexit>rLm
- θk
rL1
- θk
θk+1 = θk α X
m∈Mk
rLm(θk) α X
m∈M/Mk
rLm ˆ θ
k−1 m
- <latexit sha1_base64="xjayPFLH2d2e1FfhubKeZ30oZTE=">AC2HicbVJdixMxFM2MX2v92KqPvkSL0EW2zMiCvgLvigsILdXWzqcCdNO6FJZkjuCUM+KCIr/403/wV/gUzbcW1OzeEHM65OfmI6+UdJgkv6L4ytVr12/s3ezdun3n7n7/3v1TV9aWizEvVWnPc3BCSPGKFGJ8oK0LkSZ/nyVaufRLWydK8x1UlphoWRs4lBwxU1v/N8lLN3EqHxTMsBELz0S+fps3LmV5yEBVBbBHYbhaZ14zaTzjoOjbIDdrwUCugG7YN02mh51WB/SiWYcdcwpcsWve+1vgnz/L5WLICkDfUSjo4UCHadNmHWT9QTJK1kEvg3QLBmQbJ1n/J5uVvNbCIA/tuEmaVDj1YFyJZoeq52ogC9hISYBGtDCTf36YRr6JDAzOi9tmAbpmr24w4N2bchUwMWbldryS5tUuP8xdRLU9UoDN8UmteKYknbV6YzaQVHtQoAuJWhV8oLsMAx/IVeuIR098iXwemzUZqM0ndHg+Oj7XskYfkMRmSlDwnx+Q1OSFjwqNx5KMv0df4Q/w5/hZ/36TG0XbPA/JfxD/+AHy5xQ=</latexit><latexit sha1_base64="xjayPFLH2d2e1FfhubKeZ30oZTE=">AC2HicbVJdixMxFM2MX2v92KqPvkSL0EW2zMiCvgLvigsILdXWzqcCdNO6FJZkjuCUM+KCIr/403/wV/gUzbcW1OzeEHM65OfmI6+UdJgkv6L4ytVr12/s3ezdun3n7n7/3v1TV9aWizEvVWnPc3BCSPGKFGJ8oK0LkSZ/nyVaufRLWydK8x1UlphoWRs4lBwxU1v/N8lLN3EqHxTMsBELz0S+fps3LmV5yEBVBbBHYbhaZ14zaTzjoOjbIDdrwUCugG7YN02mh51WB/SiWYcdcwpcsWve+1vgnz/L5WLICkDfUSjo4UCHadNmHWT9QTJK1kEvg3QLBmQbJ1n/J5uVvNbCIA/tuEmaVDj1YFyJZoeq52ogC9hISYBGtDCTf36YRr6JDAzOi9tmAbpmr24w4N2bchUwMWbldryS5tUuP8xdRLU9UoDN8UmteKYknbV6YzaQVHtQoAuJWhV8oLsMAx/IVeuIR098iXwemzUZqM0ndHg+Oj7XskYfkMRmSlDwnx+Q1OSFjwqNx5KMv0df4Q/w5/hZ/36TG0XbPA/JfxD/+AHy5xQ=</latexit><latexit sha1_base64="xjayPFLH2d2e1FfhubKeZ30oZTE=">AC2HicbVJdixMxFM2MX2v92KqPvkSL0EW2zMiCvgLvigsILdXWzqcCdNO6FJZkjuCUM+KCIr/403/wV/gUzbcW1OzeEHM65OfmI6+UdJgkv6L4ytVr12/s3ezdun3n7n7/3v1TV9aWizEvVWnPc3BCSPGKFGJ8oK0LkSZ/nyVaufRLWydK8x1UlphoWRs4lBwxU1v/N8lLN3EqHxTMsBELz0S+fps3LmV5yEBVBbBHYbhaZ14zaTzjoOjbIDdrwUCugG7YN02mh51WB/SiWYcdcwpcsWve+1vgnz/L5WLICkDfUSjo4UCHadNmHWT9QTJK1kEvg3QLBmQbJ1n/J5uVvNbCIA/tuEmaVDj1YFyJZoeq52ogC9hISYBGtDCTf36YRr6JDAzOi9tmAbpmr24w4N2bchUwMWbldryS5tUuP8xdRLU9UoDN8UmteKYknbV6YzaQVHtQoAuJWhV8oLsMAx/IVeuIR098iXwemzUZqM0ndHg+Oj7XskYfkMRmSlDwnx+Q1OSFjwqNx5KMv0df4Q/w5/hZ/36TG0XbPA/JfxD/+AHy5xQ=</latexit><latexit sha1_base64="xjayPFLH2d2e1FfhubKeZ30oZTE=">AC2HicbVJdixMxFM2MX2v92KqPvkSL0EW2zMiCvgLvigsILdXWzqcCdNO6FJZkjuCUM+KCIr/403/wV/gUzbcW1OzeEHM65OfmI6+UdJgkv6L4ytVr12/s3ezdun3n7n7/3v1TV9aWizEvVWnPc3BCSPGKFGJ8oK0LkSZ/nyVaufRLWydK8x1UlphoWRs4lBwxU1v/N8lLN3EqHxTMsBELz0S+fps3LmV5yEBVBbBHYbhaZ14zaTzjoOjbIDdrwUCugG7YN02mh51WB/SiWYcdcwpcsWve+1vgnz/L5WLICkDfUSjo4UCHadNmHWT9QTJK1kEvg3QLBmQbJ1n/J5uVvNbCIA/tuEmaVDj1YFyJZoeq52ogC9hISYBGtDCTf36YRr6JDAzOi9tmAbpmr24w4N2bchUwMWbldryS5tUuP8xdRLU9UoDN8UmteKYknbV6YzaQVHtQoAuJWhV8oLsMAx/IVeuIR098iXwemzUZqM0ndHg+Oj7XskYfkMRmSlDwnx+Q1OSFjwqNx5KMv0df4Q/w5/hZ/36TG0XbPA/JfxD/+AHy5xQ=</latexit>
q Select a subset of workers to upload q Remaining workers in do not upload M/Mk
<latexit sha1_base64="qvz+PV4H383Let9dvoj1mypjLYI=">ACAnicbVDLSgMxFL1TX7W+Rl2Jm2ARXJWZUtBlwY0boYJ9QGcsmThmYyQ5IRylDc+CtuXCji1q9w59+YtiNo64HAyTn3uSeIOFMacf5sgorq2vrG8XN0tb2zu6evX/QUnEqCW2SmMeyE2BFORO0qZnmtJNIiqOA03Ywupz67XsqFYvFrR4n1I/wQLCQEayN1LOPMo9gjq4nuJYDdHP9W7Us8tOxZkBLRM3J2XI0ejZn14/JmlEhSZmluq6TqL9DEvNCKeTkpcqmAywgPaNVTgiCo/m60wQadG6aMwluYIjWbq74MR0qNo8BURlgP1aI3Ff/zuqkOL/yMiSTVJD5Q2HKkY7RNA/UZ5ISzceGYCKZ+SsiQywx0Sa1kgnBXVx5mbSqFdepuDe1cr2ax1GEYziBM3DhHOpwBQ1oAoEHeIXeLUerWfrzXqflxasvOcQ/sD6+AaVU5bX</latexit><latexit sha1_base64="qvz+PV4H383Let9dvoj1mypjLYI=">ACAnicbVDLSgMxFL1TX7W+Rl2Jm2ARXJWZUtBlwY0boYJ9QGcsmThmYyQ5IRylDc+CtuXCji1q9w59+YtiNo64HAyTn3uSeIOFMacf5sgorq2vrG8XN0tb2zu6evX/QUnEqCW2SmMeyE2BFORO0qZnmtJNIiqOA03Ywupz67XsqFYvFrR4n1I/wQLCQEayN1LOPMo9gjq4nuJYDdHP9W7Us8tOxZkBLRM3J2XI0ejZn14/JmlEhSZmluq6TqL9DEvNCKeTkpcqmAywgPaNVTgiCo/m60wQadG6aMwluYIjWbq74MR0qNo8BURlgP1aI3Ff/zuqkOL/yMiSTVJD5Q2HKkY7RNA/UZ5ISzceGYCKZ+SsiQywx0Sa1kgnBXVx5mbSqFdepuDe1cr2ax1GEYziBM3DhHOpwBQ1oAoEHeIXeLUerWfrzXqflxasvOcQ/sD6+AaVU5bX</latexit><latexit sha1_base64="qvz+PV4H383Let9dvoj1mypjLYI=">ACAnicbVDLSgMxFL1TX7W+Rl2Jm2ARXJWZUtBlwY0boYJ9QGcsmThmYyQ5IRylDc+CtuXCji1q9w59+YtiNo64HAyTn3uSeIOFMacf5sgorq2vrG8XN0tb2zu6evX/QUnEqCW2SmMeyE2BFORO0qZnmtJNIiqOA03Ywupz67XsqFYvFrR4n1I/wQLCQEayN1LOPMo9gjq4nuJYDdHP9W7Us8tOxZkBLRM3J2XI0ejZn14/JmlEhSZmluq6TqL9DEvNCKeTkpcqmAywgPaNVTgiCo/m60wQadG6aMwluYIjWbq74MR0qNo8BURlgP1aI3Ff/zuqkOL/yMiSTVJD5Q2HKkY7RNA/UZ5ISzceGYCKZ+SsiQywx0Sa1kgnBXVx5mbSqFdepuDe1cr2ax1GEYziBM3DhHOpwBQ1oAoEHeIXeLUerWfrzXqflxasvOcQ/sD6+AaVU5bX</latexit><latexit sha1_base64="qvz+PV4H383Let9dvoj1mypjLYI=">ACAnicbVDLSgMxFL1TX7W+Rl2Jm2ARXJWZUtBlwY0boYJ9QGcsmThmYyQ5IRylDc+CtuXCji1q9w59+YtiNo64HAyTn3uSeIOFMacf5sgorq2vrG8XN0tb2zu6evX/QUnEqCW2SmMeyE2BFORO0qZnmtJNIiqOA03Ywupz67XsqFYvFrR4n1I/wQLCQEayN1LOPMo9gjq4nuJYDdHP9W7Us8tOxZkBLRM3J2XI0ejZn14/JmlEhSZmluq6TqL9DEvNCKeTkpcqmAywgPaNVTgiCo/m60wQadG6aMwluYIjWbq74MR0qNo8BURlgP1aI3Ff/zuqkOL/yMiSTVJD5Q2HKkY7RNA/UZ5ISzceGYCKZ+SsiQywx0Sa1kgnBXVx5mbSqFdepuDe1cr2ax1GEYziBM3DhHOpwBQ1oAoEHeIXeLUerWfrzXqflxasvOcQ/sD6+AaVU5bX</latexit>Mk ⊆ M
<latexit sha1_base64="KL3ZLMNta0F+BxwcXvA28JXtUA=">ACBXicbVBNS8NAEN3Ur1q/oh71sFgETyUpgh4LXrwIFewHNLFstpN26WYTdzdCb148a948aCIV/+DN/+N2zaCtj4YeLw3w8y8IOFMacf5sgpLyura8X10sbm1vaOvbvXVHEqKTRozGPZDogCzgQ0NMc2okEgUcWsHwYuK37kEqFosbPUrAj0hfsJBRo3UtQ8zjxKOr8a3Q0+lgQINd/hH69plp+JMgReJm5MylHv2p9eL6ZpBEJTpTquE6i/YxIzSiHclLFSEDkfOoYKEoHys+kXY3xslB4OY2lKaDxVf09kJFJqFAWmMyJ6oOa9ifif10l1eO5nTCSpBkFni8KUYx3jS4xyRQzUeGECqZuRXTAZGEahNcyYTgzr+8SJrViutU3OvTcq2ax1FEB+gInSAXnaEaukR1EAUPaAn9IJerUfr2Xqz3metBSuf2Ud/YH18AySFmEY=</latexit><latexit sha1_base64="KL3ZLMNta0F+BxwcXvA28JXtUA=">ACBXicbVBNS8NAEN3Ur1q/oh71sFgETyUpgh4LXrwIFewHNLFstpN26WYTdzdCb148a948aCIV/+DN/+N2zaCtj4YeLw3w8y8IOFMacf5sgpLyura8X10sbm1vaOvbvXVHEqKTRozGPZDogCzgQ0NMc2okEgUcWsHwYuK37kEqFosbPUrAj0hfsJBRo3UtQ8zjxKOr8a3Q0+lgQINd/hH69plp+JMgReJm5MylHv2p9eL6ZpBEJTpTquE6i/YxIzSiHclLFSEDkfOoYKEoHys+kXY3xslB4OY2lKaDxVf09kJFJqFAWmMyJ6oOa9ifif10l1eO5nTCSpBkFni8KUYx3jS4xyRQzUeGECqZuRXTAZGEahNcyYTgzr+8SJrViutU3OvTcq2ax1FEB+gInSAXnaEaukR1EAUPaAn9IJerUfr2Xqz3metBSuf2Ud/YH18AySFmEY=</latexit><latexit sha1_base64="KL3ZLMNta0F+BxwcXvA28JXtUA=">ACBXicbVBNS8NAEN3Ur1q/oh71sFgETyUpgh4LXrwIFewHNLFstpN26WYTdzdCb148a948aCIV/+DN/+N2zaCtj4YeLw3w8y8IOFMacf5sgpLyura8X10sbm1vaOvbvXVHEqKTRozGPZDogCzgQ0NMc2okEgUcWsHwYuK37kEqFosbPUrAj0hfsJBRo3UtQ8zjxKOr8a3Q0+lgQINd/hH69plp+JMgReJm5MylHv2p9eL6ZpBEJTpTquE6i/YxIzSiHclLFSEDkfOoYKEoHys+kXY3xslB4OY2lKaDxVf09kJFJqFAWmMyJ6oOa9ifif10l1eO5nTCSpBkFni8KUYx3jS4xyRQzUeGECqZuRXTAZGEahNcyYTgzr+8SJrViutU3OvTcq2ax1FEB+gInSAXnaEaukR1EAUPaAn9IJerUfr2Xqz3metBSuf2Ud/YH18AySFmEY=</latexit><latexit sha1_base64="KL3ZLMNta0F+BxwcXvA28JXtUA=">ACBXicbVBNS8NAEN3Ur1q/oh71sFgETyUpgh4LXrwIFewHNLFstpN26WYTdzdCb148a948aCIV/+DN/+N2zaCtj4YeLw3w8y8IOFMacf5sgpLyura8X10sbm1vaOvbvXVHEqKTRozGPZDogCzgQ0NMc2okEgUcWsHwYuK37kEqFosbPUrAj0hfsJBRo3UtQ8zjxKOr8a3Q0+lgQINd/hH69plp+JMgReJm5MylHv2p9eL6ZpBEJTpTquE6i/YxIzSiHclLFSEDkfOoYKEoHys+kXY3xslB4OY2lKaDxVf09kJFJqFAWmMyJ6oOa9ifif10l1eO5nTCSpBkFni8KUYx3jS4xyRQzUeGECqZuRXTAZGEahNcyYTgzr+8SJrViutU3OvTcq2ax1FEB+gInSAXnaEaukR1EAUPaAn9IJerUfr2Xqz3metBSuf2Ud/YH18AySFmEY=</latexit>Old gradient Fresh gradient
7
LAG: GD under two alternative communication rules
- T. Chen, G. B. Giannakis, T. Sun, and W. Yin "LAG: Lazily Aggregated Gradient for Communication-Efficient
Distributed Learning," Proc. of NIPS, Montreal, Canada, December 3-8, 2018.
q Worker-side rule (LAG-WK): Include worker m in if
Gradient innovation Optimization progress
- rLm
- θk
rLm ˆ θ
k−1 m
- 1
M
- 1
α ⇣ θk θk−1⌘
- <latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="0R4hPLkwHklCrgKU4UNeGnfaJvs=">ACw3icbVJNb9NAEF2brxIKBK5cVlRI7SGRzQWOC5IgBQk0lbKptF4M7ZXWX90d1wpbC3+Izd+AP+DdRwkaD3Sap/em9HMvtmk1spSFP0Kwjt3791/cPBw9Ojw8ZOn42eHp7ZqjMS5rHRlzhOwqFWJc1Kk8bw2CEWi8SzZfOj0sys0VlXlN9rWuCwgK1WqJCnVuPf4r3KxDUXJSQanJCg+ed2VYhEZciqfTabgt/OUE5ErQXm045mQzn50BuoMjrF24zidtdbd9RZHjJRWpAurh1X1qhMSVP/2UE6DqHnh4eZDLE9m2MynI6S9xPRqtxkfRNoFvw3iPThi+5itxj/FupJNgSVJDdYu4qimpQNDSmpsR6KxWIPcQIYLD0so0C7dbh0tf+WZNU8r409JfMf+W+GgsN3YPrMAyu1NrSOHtEVD6dulU2XdEJayb5Q2mlPFu93ytTIoSW89AGmUn5XLHLyh5H9AZ0J8m3wenraRxN468RO2Av2Et2zGL2hr1jH9mMzZkMPgWXwfAhbPwKmx7u8Jg79tz9l+EP/4AYCLkSg=</latexit><latexit sha1_base64="0R4hPLkwHklCrgKU4UNeGnfaJvs=">ACw3icbVJNb9NAEF2brxIKBK5cVlRI7SGRzQWOC5IgBQk0lbKptF4M7ZXWX90d1wpbC3+Izd+AP+DdRwkaD3Sap/em9HMvtmk1spSFP0Kwjt3791/cPBw9Ojw8ZOn42eHp7ZqjMS5rHRlzhOwqFWJc1Kk8bw2CEWi8SzZfOj0sys0VlXlN9rWuCwgK1WqJCnVuPf4r3KxDUXJSQanJCg+ed2VYhEZciqfTabgt/OUE5ErQXm045mQzn50BuoMjrF24zidtdbd9RZHjJRWpAurh1X1qhMSVP/2UE6DqHnh4eZDLE9m2MynI6S9xPRqtxkfRNoFvw3iPThi+5itxj/FupJNgSVJDdYu4qimpQNDSmpsR6KxWIPcQIYLD0so0C7dbh0tf+WZNU8r409JfMf+W+GgsN3YPrMAyu1NrSOHtEVD6dulU2XdEJayb5Q2mlPFu93ytTIoSW89AGmUn5XLHLyh5H9AZ0J8m3wenraRxN468RO2Av2Et2zGL2hr1jH9mMzZkMPgWXwfAhbPwKmx7u8Jg79tz9l+EP/4AYCLkSg=</latexit><latexit sha1_base64="CkJErpo0e1xvgGRANXgvbPzUtg=">ACznicbVJNa9wEJXdr3T7tWmPvYguheSwi51LegzNpdAWtBNAqvNMtbKtlhZdqRxYKuYXv7eusP6P+ovN5Cm3hA6PHePGY0o6RS0mIU/QrCe/cfPHy093jw5Omz5y+G+y/PbFkbLma8VKW5SMAKJbWYoUQlLiojoEiUOE/Wp61+fi2MlaX+iptKLArItEwlB/TUcvibvZcZu6FMQ6LAMQ6KfmqWBUtkdsCSUq3spvCXY5gLhOZy3SqH4/78HND1mLx+6dbjuNl6u4osE1eUpQa4ixv3uWFKpOjpvwDVeXQ0f2NjPvYroyRWY6H3cVuBoPlcBRNom3QuyDegRHZxXQ5/MlWJa8LoZErsHYeRxUuHBiUXIlmwGorKuBryMTcQw2FsAu3XUdD3pmRdPS+KORbtl/HQ4K27btMwvA3N7WrJPm9eYvls4qasaheZdobRWFEva7paupBEc1cYD4Eb6XinPwQ8U/Q9ohxDfvJdcHY0iaNJ/CUanRztxrFHXpM35IDE5JickA9kSmaEBx+Dq+Bb4MJpeB024fcuNQx2nlfkvwh/AFHReVw</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit>
Old gradient
8
LAG: GD under two alternative communication rules
- T. Chen, G. B. Giannakis, T. Sun, and W. Yin "LAG: Lazily Aggregated Gradient for Communication-Efficient
Distributed Learning," Proc. of NIPS, Montreal, Canada, December 3-8, 2018.
q Worker-side rule (LAG-WK): q Server-side rule (LAG-PS):
: smoothness of
Include worker m in if § LAG-PS is a sufficient condition for LAG-WK.
Lm
<latexit sha1_base64="IsVjkbiGRCOxdOt70vZudkTBfj4=">AB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJgY2ERwXxgcoS9zSRZsrt37O4J4ci/sLFQxNZ/Y+e/cZNcoYkPBh7vzTAzL0oEN9b3v73C2vrG5lZxu7Szu7d/UD48apo41QwbLBaxbkfUoOAKG5Zbge1EI5WRwFY0vpn5rSfUhsfqwU4SDCUdKj7gjFonPWZdRgW5m/Zkr1zxq/4cZJUEOalAjnqv/NXtxyVqCwT1JhO4Cc2zKi2nAmclrqpwYSyMR1ix1FJZowm18JWdO6ZNBrF0pS+bq74mMSmMmMnKdktqRWfZm4n9eJ7WD6zDjKktKrZYNEgFsTGZvU/6XCOzYuIZq7WwkbU2ZdSGVXAjB8surpHlRDfxqcH9ZqV3mcRThBE7hHAK4ghrcQh0awEDBM7zCm2e8F+/d+1i0Frx85hj+wPv8ATLPkIw=</latexit><latexit sha1_base64="IsVjkbiGRCOxdOt70vZudkTBfj4=">AB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJgY2ERwXxgcoS9zSRZsrt37O4J4ci/sLFQxNZ/Y+e/cZNcoYkPBh7vzTAzL0oEN9b3v73C2vrG5lZxu7Szu7d/UD48apo41QwbLBaxbkfUoOAKG5Zbge1EI5WRwFY0vpn5rSfUhsfqwU4SDCUdKj7gjFonPWZdRgW5m/Zkr1zxq/4cZJUEOalAjnqv/NXtxyVqCwT1JhO4Cc2zKi2nAmclrqpwYSyMR1ix1FJZowm18JWdO6ZNBrF0pS+bq74mMSmMmMnKdktqRWfZm4n9eJ7WD6zDjKktKrZYNEgFsTGZvU/6XCOzYuIZq7WwkbU2ZdSGVXAjB8surpHlRDfxqcH9ZqV3mcRThBE7hHAK4ghrcQh0awEDBM7zCm2e8F+/d+1i0Frx85hj+wPv8ATLPkIw=</latexit><latexit sha1_base64="IsVjkbiGRCOxdOt70vZudkTBfj4=">AB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJgY2ERwXxgcoS9zSRZsrt37O4J4ci/sLFQxNZ/Y+e/cZNcoYkPBh7vzTAzL0oEN9b3v73C2vrG5lZxu7Szu7d/UD48apo41QwbLBaxbkfUoOAKG5Zbge1EI5WRwFY0vpn5rSfUhsfqwU4SDCUdKj7gjFonPWZdRgW5m/Zkr1zxq/4cZJUEOalAjnqv/NXtxyVqCwT1JhO4Cc2zKi2nAmclrqpwYSyMR1ix1FJZowm18JWdO6ZNBrF0pS+bq74mMSmMmMnKdktqRWfZm4n9eJ7WD6zDjKktKrZYNEgFsTGZvU/6XCOzYuIZq7WwkbU2ZdSGVXAjB8surpHlRDfxqcH9ZqV3mcRThBE7hHAK4ghrcQh0awEDBM7zCm2e8F+/d+1i0Frx85hj+wPv8ATLPkIw=</latexit><latexit sha1_base64="IsVjkbiGRCOxdOt70vZudkTBfj4=">AB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5E0DJgY2ERwXxgcoS9zSRZsrt37O4J4ci/sLFQxNZ/Y+e/cZNcoYkPBh7vzTAzL0oEN9b3v73C2vrG5lZxu7Szu7d/UD48apo41QwbLBaxbkfUoOAKG5Zbge1EI5WRwFY0vpn5rSfUhsfqwU4SDCUdKj7gjFonPWZdRgW5m/Zkr1zxq/4cZJUEOalAjnqv/NXtxyVqCwT1JhO4Cc2zKi2nAmclrqpwYSyMR1ix1FJZowm18JWdO6ZNBrF0pS+bq74mMSmMmMnKdktqRWfZm4n9eJ7WD6zDjKktKrZYNEgFsTGZvU/6XCOzYuIZq7WwkbU2ZdSGVXAjB8surpHlRDfxqcH9ZqV3mcRThBE7hHAK4ghrcQh0awEDBM7zCm2e8F+/d+1i0Frx85hj+wPv8ATLPkIw=</latexit>Include worker m in if
Gradient innovation Optimization progress
- rLm
- θk
rLm ˆ θ
k−1 m
- 1
M
- 1
α ⇣ θk θk−1⌘
- <latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="0R4hPLkwHklCrgKU4UNeGnfaJvs=">ACw3icbVJNb9NAEF2brxIKBK5cVlRI7SGRzQWOC5IgBQk0lbKptF4M7ZXWX90d1wpbC3+Izd+AP+DdRwkaD3Sap/em9HMvtmk1spSFP0Kwjt3791/cPBw9Ojw8ZOn42eHp7ZqjMS5rHRlzhOwqFWJc1Kk8bw2CEWi8SzZfOj0sys0VlXlN9rWuCwgK1WqJCnVuPf4r3KxDUXJSQanJCg+ed2VYhEZciqfTabgt/OUE5ErQXm045mQzn50BuoMjrF24zidtdbd9RZHjJRWpAurh1X1qhMSVP/2UE6DqHnh4eZDLE9m2MynI6S9xPRqtxkfRNoFvw3iPThi+5itxj/FupJNgSVJDdYu4qimpQNDSmpsR6KxWIPcQIYLD0so0C7dbh0tf+WZNU8r409JfMf+W+GgsN3YPrMAyu1NrSOHtEVD6dulU2XdEJayb5Q2mlPFu93ytTIoSW89AGmUn5XLHLyh5H9AZ0J8m3wenraRxN468RO2Av2Et2zGL2hr1jH9mMzZkMPgWXwfAhbPwKmx7u8Jg79tz9l+EP/4AYCLkSg=</latexit><latexit sha1_base64="0R4hPLkwHklCrgKU4UNeGnfaJvs=">ACw3icbVJNb9NAEF2brxIKBK5cVlRI7SGRzQWOC5IgBQk0lbKptF4M7ZXWX90d1wpbC3+Izd+AP+DdRwkaD3Sap/em9HMvtmk1spSFP0Kwjt3791/cPBw9Ojw8ZOn42eHp7ZqjMS5rHRlzhOwqFWJc1Kk8bw2CEWi8SzZfOj0sys0VlXlN9rWuCwgK1WqJCnVuPf4r3KxDUXJSQanJCg+ed2VYhEZciqfTabgt/OUE5ErQXm045mQzn50BuoMjrF24zidtdbd9RZHjJRWpAurh1X1qhMSVP/2UE6DqHnh4eZDLE9m2MynI6S9xPRqtxkfRNoFvw3iPThi+5itxj/FupJNgSVJDdYu4qimpQNDSmpsR6KxWIPcQIYLD0so0C7dbh0tf+WZNU8r409JfMf+W+GgsN3YPrMAyu1NrSOHtEVD6dulU2XdEJayb5Q2mlPFu93ytTIoSW89AGmUn5XLHLyh5H9AZ0J8m3wenraRxN468RO2Av2Et2zGL2hr1jH9mMzZkMPgWXwfAhbPwKmx7u8Jg79tz9l+EP/4AYCLkSg=</latexit><latexit sha1_base64="CkJErpo0e1xvgGRANXgvbPzUtg=">ACznicbVJNa9wEJXdr3T7tWmPvYguheSwi51LegzNpdAWtBNAqvNMtbKtlhZdqRxYKuYXv7eusP6P+ovN5Cm3hA6PHePGY0o6RS0mIU/QrCe/cfPHy093jw5Omz5y+G+y/PbFkbLma8VKW5SMAKJbWYoUQlLiojoEiUOE/Wp61+fi2MlaX+iptKLArItEwlB/TUcvibvZcZu6FMQ6LAMQ6KfmqWBUtkdsCSUq3spvCXY5gLhOZy3SqH4/78HND1mLx+6dbjuNl6u4osE1eUpQa4ixv3uWFKpOjpvwDVeXQ0f2NjPvYroyRWY6H3cVuBoPlcBRNom3QuyDegRHZxXQ5/MlWJa8LoZErsHYeRxUuHBiUXIlmwGorKuBryMTcQw2FsAu3XUdD3pmRdPS+KORbtl/HQ4K27btMwvA3N7WrJPm9eYvls4qasaheZdobRWFEva7paupBEc1cYD4Eb6XinPwQ8U/Q9ohxDfvJdcHY0iaNJ/CUanRztxrFHXpM35IDE5JickA9kSmaEBx+Dq+Bb4MJpeB024fcuNQx2nlfkvwh/AFHReVw</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit><latexit sha1_base64="sdBQ5wuLWPdC8PW6kv0JAW7GfI8=">ACznicbVJNa9wEJXdj6Tbr217EV0KSHXexQaI8hvRTawha6SWC1WcZa2RYry40DmwV02t/X2/9Af0fldbaBMPCD3em8eMZpRUSlqMol9BeOfuvft7+w8GDx89fvJ0+Oz5qS1rw8WMl6o05wlYoaQWM5SoxHlBSJEmfJ+n2rn10JY2Wpv+KmEosCMi1TyQE9tRz+ZicyY9eUaUgUOMZB0U/NsmCJzA5YUqV3RT+cgxzgdBcrFvlcNyfnwO6HpPXL9x6HDdb1eRZeKSstQAd3HjPjdMiRQ9/ZdhoKocOrq/kXEf25UxMsvxsLvY9WCwHI6iSbQNehvEOzAiu5guhz/ZquR1ITRyBdbO46jChQODkivRDFhtRQV8DZmYe6ihEHbhtuto6GvPrGhaGn80i37r8NBYdu2fWYBmNubWkv2afMa03cLJ3Vo9C8K5TWimJ293SlTSCo9p4ANxI3yvlOfiBov8B7RDim0+DU6PJnE0ib+8GR0f7caxT16SV+SAxOQtOSYfyJTMCA8+BpfBt8CF0/AqbMLvXWoY7DwvyH8R/vgDSIXldA=</latexit>
Lm
- θk − ˆ
θ
k−1 m
- ≥ 1
M
- 1
α ⇣ θk − θk−1⌘
- <latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="EXy2rVIBegRSOrMj8+MSFv1Xw2g=">ACknicdZHPbtQwEMad8K8sBRaOcMCiQiqHrpJe4FiBkAVCS2rbTeRhPvJLHWToI9QVqleSuehBtvg7NZJGiXkSx/+o1HM/4mrbVyFEW/gvDGzVu37+zcHd3bvf/g4fjR7qmrGitxKitd2fMUHGpV4pQUaTyvLYJNZ6ly7d9/uw7Wqeq8iutapwbyEuVKQnkUTL+cZwY8Ubl4pKLtNILtzL+agUVSNBdLA9EAdRuSXWJuWiXB3E3VIscv3GRWZBt3LWfOqExI4/EAG6LmDA+/ptIUOHazKC3o5XOJyNErGe9EkWge/LuKN2GObOEnGP8Wiko3BkqQG52ZxVNO8BUtKauxGonFYg1xCjMvSzDo5u3a3Y6/8GTBs8r6UxJf078rWjCuH9u/NECFu5r4bcrKHs9bxVZd0QlnJolDWaU8X7VfGFsihJr7wAaZWflcsCvKHkF9qbEF/98nVxejiJo0n8JWI7Cl7zvZzF6xI/aenbApk8GT4Cj4EHwMn4XvwuPBrjDY+PaY/RPh598qd9Cj</latexit><latexit sha1_base64="EXy2rVIBegRSOrMj8+MSFv1Xw2g=">ACknicdZHPbtQwEMad8K8sBRaOcMCiQiqHrpJe4FiBkAVCS2rbTeRhPvJLHWToI9QVqleSuehBtvg7NZJGiXkSx/+o1HM/4mrbVyFEW/gvDGzVu37+zcHd3bvf/g4fjR7qmrGitxKitd2fMUHGpV4pQUaTyvLYJNZ6ly7d9/uw7Wqeq8iutapwbyEuVKQnkUTL+cZwY8Ubl4pKLtNILtzL+agUVSNBdLA9EAdRuSXWJuWiXB3E3VIscv3GRWZBt3LWfOqExI4/EAG6LmDA+/ptIUOHazKC3o5XOJyNErGe9EkWge/LuKN2GObOEnGP8Wiko3BkqQG52ZxVNO8BUtKauxGonFYg1xCjMvSzDo5u3a3Y6/8GTBs8r6UxJf078rWjCuH9u/NECFu5r4bcrKHs9bxVZd0QlnJolDWaU8X7VfGFsihJr7wAaZWflcsCvKHkF9qbEF/98nVxejiJo0n8JWI7Cl7zvZzF6xI/aenbApk8GT4Cj4EHwMn4XvwuPBrjDY+PaY/RPh598qd9Cj</latexit><latexit sha1_base64="FwqDNGX8HsaSJO47Ghwur+5XsMo=">ACnXicdVFNj9MwEHXC1K+unCDAxYV0nLYKtkLHCsQEqAFLRLdrlR3o4nrJFbtJGtPkCpv/hW/hBv/BqcpEuyWkax5em9GM36T1kpajKJfQXj5q3bd/buDu7df/Dw0XD/8amtGsPFlFeqMmcpWKFkKaYoUYmz2gjQqRKzdPWu02fhbGyKr/huhYLDXkpM8kBPZUMfxwnmr2VObukLK3U0q61T45hIRDa89UhKwDdDqlN9LlbHcZt381ycUFZoC7uHWfW6ZEhp7+wzBQdQE9fCfSTvYfoKReYGv+sQuB4NkOIrG0SbodRBvwYhs4yQZ/mTLijdalMgVWDuPoxoXDgxKrkQ7YI0VNfAV5GLuYQla2IXbuNvSl5Z0qwy/pVIN+zfHQ607db2lRqwsFe1jtylzRvM3iycLOsGRcn7QVmjKFa0OxVdSiM4qrUHwI30u1JegDcU/UE7E+KrX74OTo/GcTSOv0ajydHWj3yjLwgByQmr8mEfCAnZEp48DSYB+DT+Hz8H14H7pS8Ng2/OE/BPh7DdAEdGY</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit><latexit sha1_base64="YZvfKo7wYha/YERVZH/rng1YfkA=">ACnXicdVFdi9NAFJ3Er7V+bNU3fXCwCOvDlmQR9LEogsoqK9jtQqcbJtObZOhMEmduhDKbf+Uv8c1/46SpoLv1wnAP59zLvXNuWitpMYp+BeG16zdu3tq7Pbhz979/eGDh6e2aoyAqahUZc5SbkHJEqYoUcFZbYDrVMEsXb3t9Nl3MFZW5Vdc17DQPC9lJgVHTyXDH8eJZm9kzi4oSyu1tGvtk2NYAPL2fHXICo5uh9Qm+tytDuO272Y5fKMsM1y4uHWfWqYgQ0/YRhXdcF7+uA/k3aw/Qj8wJf9IldDAbJcBSNo03QqyDeghHZxky/MmWlWg0lCgUt3YeRzUuHDcohYJ2wBoLNRcrnsPcw5JrsAu3cbelz2zpFl/CuRbti/OxzXtlvbV2qOhb2sdeQubd5g9nrhZFk3CKXoB2WNoljR7lR0KQ0IVGsPuDS70pFwb2h6A/amRBf/vJVcHo0jqNx/OXlaHK0tWOPCHPyAGJySsyIe/JCZkSETwOJsGH4GP4NHwXHoef+9Iw2PY8Iv9EOPsNQVHRnA=</latexit>
Old gradient
9
Iteration and communication complexity
(nonconvex) Local loss is smooth. (convex) Loss is convex. (strongly convex) Loss is (restricted) strongly convex. In all cases, LAG enjoys the same convergence rate as GD.
Theorem 1
If local objectives are heterogeneous, LAG requires smaller number of uploads to a given accuracy than GD; e.g., as small as 1/M.
Theorem 2
10
Linear prediction
q Real datasets distributed on M = 9 workers
- M. Lichman, “UCI machine learning repository,” 2013. [Online]: http://archive.ics.uci.edu/ml