Closing the convergence gap of SGD without replacement Shashank - - PowerPoint PPT Presentation

closing the convergence gap of sgd without replacement
SMART_READER_LITE
LIVE PREVIEW

Closing the convergence gap of SGD without replacement Shashank - - PowerPoint PPT Presentation

Closing the convergence gap of SGD without replacement Shashank Rajput, Anant Gupta, Dimitris Papailiopoulos <latexit


slide-1
SLIDE 1

Closing the convergence gap of SGD without replacement

Shashank Rajput, Anant Gupta, Dimitris Papailiopoulos

slide-2
SLIDE 2

Stochastic Gradient Descent (SGD)

  • Algorithm:

1. 2. 3. Repeat for T iterations

  • SGD with replacement is theoretically well understood

Sampling with replacement

However, in practice we sample without replacement

xt+1 := xt αrfi(xt), α is the step size

<latexit sha1_base64="3fen9BguvaKkANsAF6iVMK8pUE=">ACMHicbZDfaxNBEMfnqtWaVo0/3nwZDEJFDXciKIQ6IN9rGDaQhLOuc1cs3Rv79idk8Qjf5Iv/in6UkEpvpXuEmK2NYvLHz5zAyz80qo73E8fdo7crV9WvXN260Nrdu3rdvnN35e1U9xXpSndYUaejbcFy2GDyvHVGSGD7LjnUX94CM7r0v7XmYVjwo6sjrXiSgtP12mjbyJnj6zfTVPAZDslUE8KhpcwQ5qneDvzx079ceCoNao8yYfTCFXr9iedpuxN346XwsknOTKd3P8/AMBe2v46HJeqLtiKMuT9IkrGTXkRCvD89aw9lyROqYjHgRrqWA/apYHz/FRIGPMSxeFVzSfycaKryfFVnoLEgm/mJtAf9XG9Svxo12la1sFWrRXltUEpcpIdj7ViJmQVDyunwV1QTcqQkZNwKISQXT75s9p93k7ibvEs6vRew0gY8gIewDQm8hB7swh70QcFn+AY/4Gf0JTqJTqNfq9a16GzmHpxT9PsPibipjw=</latexit><latexit sha1_base64="Kow3Z1P1X7aZhSDItpuefcsQw2I=">ACMHicbZDfSxtBEMf3tLa1pa3/oyNBQU23BXChVBCPhQHy0GkjCMbeZM4t7e8fuXEk8if54p9SXxRaiq/9K7r5QWm1X1j48pkZuebFo5DsPbYGn50crjJ6trtafP1p9v1F9snri8tJLaMte57SToSCtDbVasqVNYwizRdJqcH07rp1/JOpWbLzwuqJ/hmVGpksgexfVPo7ji3WgC+wejmOEd9FAXQ4SewUQjpLHa9nzn7R/ONOIKlAMeEjimApy6oElcb4TNcCZ4aKFabS20pmO4/q3iCXZUaGpUbnulFYcL9Cy0pqmtR6paMC5TmeUdbgxm5fjU7eAJvPBlAmlv/DMOM/j1RYebcOEt8Z4Y8dPdrU/i/WrfkdK9fKVOUTEbOF6WlBs5hmh4MlCXJeuwNSqv8X0EO0aJkn3HNhxDdP/mhOXnfjMJm9DlqtD6IuVbFK/FabItIfBQtcSORVtIcSmuxXfxI7gKboKfwd28dSlYzLwU/yj49RsqsqrH</latexit><latexit sha1_base64="Kow3Z1P1X7aZhSDItpuefcsQw2I=">ACMHicbZDfSxtBEMf3tLa1pa3/oyNBQU23BXChVBCPhQHy0GkjCMbeZM4t7e8fuXEk8if54p9SXxRaiq/9K7r5QWm1X1j48pkZuebFo5DsPbYGn50crjJ6trtafP1p9v1F9snri8tJLaMte57SToSCtDbVasqVNYwizRdJqcH07rp1/JOpWbLzwuqJ/hmVGpksgexfVPo7ji3WgC+wejmOEd9FAXQ4SewUQjpLHa9nzn7R/ONOIKlAMeEjimApy6oElcb4TNcCZ4aKFabS20pmO4/q3iCXZUaGpUbnulFYcL9Cy0pqmtR6paMC5TmeUdbgxm5fjU7eAJvPBlAmlv/DMOM/j1RYebcOEt8Z4Y8dPdrU/i/WrfkdK9fKVOUTEbOF6WlBs5hmh4MlCXJeuwNSqv8X0EO0aJkn3HNhxDdP/mhOXnfjMJm9DlqtD6IuVbFK/FabItIfBQtcSORVtIcSmuxXfxI7gKboKfwd28dSlYzLwU/yj49RsqsqrH</latexit><latexit sha1_base64="4Npo/c1Rtmo0CPzKk2Fd+1l0KI0=">ACMHicbZBRaxNBEMf3WltjWtuoj74MhkJLa7grgiIAR/0sYJpC0k45jZzZK9vWN3ThKPfKS+9KPUF4UW8dVP4SYNokn/sPDnNzPMzj8ptHIchj+CtfUHG5sPa4/qW9uPd3YbT56eury0kjoy17k9T9CRVoY6rFjTeWEJs0TWTJ6P6ufSHrVG4+86SgfoYXRqVKInsUNz6M4oPoym8fTeOGV5CD3UxROgZTDRCGqt9zw+O/nKmMVegHPCQwDEV4NRXmsaNZtgK54JVEy1MUyx0Ejeue4NclhkZlhqd60Zhwf0KLSupaVrvlY4KlCO8oK63BjNy/Wp+8BT2PBlAmlv/DMOc/jtRYebcJEt8Z4Y8dMu1Gbyv1i05fdOvlClKJiPvFqWlBs5hlh4MlCXJeuINSqv8X0EO0aJkn3HdhxAtn7xqTo9bUdiKPkXN9qtFHDXxXLwQ+yISr0VbfBQnoiOkuBTfxI24Da6C78HP4Nd61qwmHkm/lPw+w+DLKgP</latexit>

Problem : min

x

F(x) := 1

n

Pn

i=1 fi(x)

<latexit sha1_base64="Mwv+2sFCgxVJoPAQ2pdYIYCMGM=">ACLHicbVDLSgMxFM34rPVdekmWIW6KTNulEqhUBCXFWwrdOqQSTM2NMkMSUZawnyQG39FEBcWcet3mD4Waj0QOJxzLjf3hAmjSrvu2FlaXldW89t5De3tnd2C3v7LRWnEpMmjlks70KkCKOCNDXVjNwlkiAeMtIOB/WJ34kUtFY3OpRQrocPQgaUYy0lYJCvSFjm+WwAo/9VPRslGgzIzPqcj8y6vS8LRS9SOJsPEyYyWV8sDQqpfdiyig1j4OCkW37E4BF4k3J0UwRyMovPq9GKecCI0ZUqrjuYnuGiQ1xYxkeT9VJEF4gB5Ix1KBOFdMz02gydW6cEolvYJDafqzwmDuFIjHtokR7qv/noT8T+vk+romuoSFJNBJ4tilIGdQwnzcEelQRrNrIEYUntXyHuI1uMtqXlbQne35MXSeus7Ll7+asWLuY15EDh+AIlIAHzkENXIMGaAIMnsALeAdj59l5cz6cz1l0yZnPHIBfcL6+AdJKp+Q=</latexit><latexit sha1_base64="Mwv+2sFCgxVJoPAQ2pdYIYCMGM=">ACLHicbVDLSgMxFM34rPVdekmWIW6KTNulEqhUBCXFWwrdOqQSTM2NMkMSUZawnyQG39FEBcWcet3mD4Waj0QOJxzLjf3hAmjSrvu2FlaXldW89t5De3tnd2C3v7LRWnEpMmjlks70KkCKOCNDXVjNwlkiAeMtIOB/WJ34kUtFY3OpRQrocPQgaUYy0lYJCvSFjm+WwAo/9VPRslGgzIzPqcj8y6vS8LRS9SOJsPEyYyWV8sDQqpfdiyig1j4OCkW37E4BF4k3J0UwRyMovPq9GKecCI0ZUqrjuYnuGiQ1xYxkeT9VJEF4gB5Ix1KBOFdMz02gydW6cEolvYJDafqzwmDuFIjHtokR7qv/noT8T+vk+romuoSFJNBJ4tilIGdQwnzcEelQRrNrIEYUntXyHuI1uMtqXlbQne35MXSeus7Ll7+asWLuY15EDh+AIlIAHzkENXIMGaAIMnsALeAdj59l5cz6cz1l0yZnPHIBfcL6+AdJKp+Q=</latexit><latexit sha1_base64="Mwv+2sFCgxVJoPAQ2pdYIYCMGM=">ACLHicbVDLSgMxFM34rPVdekmWIW6KTNulEqhUBCXFWwrdOqQSTM2NMkMSUZawnyQG39FEBcWcet3mD4Waj0QOJxzLjf3hAmjSrvu2FlaXldW89t5De3tnd2C3v7LRWnEpMmjlks70KkCKOCNDXVjNwlkiAeMtIOB/WJ34kUtFY3OpRQrocPQgaUYy0lYJCvSFjm+WwAo/9VPRslGgzIzPqcj8y6vS8LRS9SOJsPEyYyWV8sDQqpfdiyig1j4OCkW37E4BF4k3J0UwRyMovPq9GKecCI0ZUqrjuYnuGiQ1xYxkeT9VJEF4gB5Ix1KBOFdMz02gydW6cEolvYJDafqzwmDuFIjHtokR7qv/noT8T+vk+romuoSFJNBJ4tilIGdQwnzcEelQRrNrIEYUntXyHuI1uMtqXlbQne35MXSeus7Ll7+asWLuY15EDh+AIlIAHzkENXIMGaAIMnsALeAdj59l5cz6cz1l0yZnPHIBfcL6+AdJKp+Q=</latexit><latexit sha1_base64="Mwv+2sFCgxVJoPAQ2pdYIYCMGM=">ACLHicbVDLSgMxFM34rPVdekmWIW6KTNulEqhUBCXFWwrdOqQSTM2NMkMSUZawnyQG39FEBcWcet3mD4Waj0QOJxzLjf3hAmjSrvu2FlaXldW89t5De3tnd2C3v7LRWnEpMmjlks70KkCKOCNDXVjNwlkiAeMtIOB/WJ34kUtFY3OpRQrocPQgaUYy0lYJCvSFjm+WwAo/9VPRslGgzIzPqcj8y6vS8LRS9SOJsPEyYyWV8sDQqpfdiyig1j4OCkW37E4BF4k3J0UwRyMovPq9GKecCI0ZUqrjuYnuGiQ1xYxkeT9VJEF4gB5Ix1KBOFdMz02gydW6cEolvYJDafqzwmDuFIjHtokR7qv/noT8T+vk+romuoSFJNBJ4tilIGdQwnzcEelQRrNrIEYUntXyHuI1uMtqXlbQne35MXSeus7Ll7+asWLuY15EDh+AIlIAHzkENXIMGaAIMnsALeAdj59l5cz6cz1l0yZnPHIBfcL6+AdJKp+Q=</latexit>

At each iteration, sample fi randomly from {f1, . . . , fn}

<latexit sha1_base64="bFz0n2P5KY7YuTtmLx+ZHinFcPo=">ACKnicdVDLihNBFK0eX2N8RV26uZgILkJT3Y6ZcTeDG5cjmJmBJDS3q29PiqlHU1UthCbf48ZfcTMLZXDrh1jJRFDRAwWHc+7h1j1lo6QPnF8lOzdu3rp9Z/du7979Bw8f9R8/OfG2dYImwirzkr0pKShSZB0VnjCHWp6LS8eLv2Tz+S89KaD2HZ0FzjuZG1FBiVPSPjgIQigXIQG6jcCjbhTBsC7kEByaymq1hNpZDcNZVxfZaFbZ4Ed1YWarYdEf8JS/zrM9DjzNx2/yg3Ek41f7Gc8hS/kGA7bFcdG/jHRajJBKPR+mvEmzDt0QpFq96s9dSguMBzmkZqUJOfd5tTV/AiKhXU1sVnAmzU3xMdau+XuoyTGsPC/+2txX950zbUB/NOmqYNZMT1orpVECyse4NKOhIh9lBJFE7Gv4JYoEMRe/O9WMKvS+H/5CRPM5m7/PB4d62jl32jD1nL1nG9tkhe8eO2YQJ9ol9YV/Zt+RzcplcJd+vR3eSbeYp+wPJj5+oTqYn</latexit><latexit sha1_base64="bFz0n2P5KY7YuTtmLx+ZHinFcPo=">ACKnicdVDLihNBFK0eX2N8RV26uZgILkJT3Y6ZcTeDG5cjmJmBJDS3q29PiqlHU1UthCbf48ZfcTMLZXDrh1jJRFDRAwWHc+7h1j1lo6QPnF8lOzdu3rp9Z/du7979Bw8f9R8/OfG2dYImwirzkr0pKShSZB0VnjCHWp6LS8eLv2Tz+S89KaD2HZ0FzjuZG1FBiVPSPjgIQigXIQG6jcCjbhTBsC7kEByaymq1hNpZDcNZVxfZaFbZ4Ed1YWarYdEf8JS/zrM9DjzNx2/yg3Ek41f7Gc8hS/kGA7bFcdG/jHRajJBKPR+mvEmzDt0QpFq96s9dSguMBzmkZqUJOfd5tTV/AiKhXU1sVnAmzU3xMdau+XuoyTGsPC/+2txX950zbUB/NOmqYNZMT1orpVECyse4NKOhIh9lBJFE7Gv4JYoEMRe/O9WMKvS+H/5CRPM5m7/PB4d62jl32jD1nL1nG9tkhe8eO2YQJ9ol9YV/Zt+RzcplcJd+vR3eSbeYp+wPJj5+oTqYn</latexit><latexit sha1_base64="bFz0n2P5KY7YuTtmLx+ZHinFcPo=">ACKnicdVDLihNBFK0eX2N8RV26uZgILkJT3Y6ZcTeDG5cjmJmBJDS3q29PiqlHU1UthCbf48ZfcTMLZXDrh1jJRFDRAwWHc+7h1j1lo6QPnF8lOzdu3rp9Z/du7979Bw8f9R8/OfG2dYImwirzkr0pKShSZB0VnjCHWp6LS8eLv2Tz+S89KaD2HZ0FzjuZG1FBiVPSPjgIQigXIQG6jcCjbhTBsC7kEByaymq1hNpZDcNZVxfZaFbZ4Ed1YWarYdEf8JS/zrM9DjzNx2/yg3Ek41f7Gc8hS/kGA7bFcdG/jHRajJBKPR+mvEmzDt0QpFq96s9dSguMBzmkZqUJOfd5tTV/AiKhXU1sVnAmzU3xMdau+XuoyTGsPC/+2txX950zbUB/NOmqYNZMT1orpVECyse4NKOhIh9lBJFE7Gv4JYoEMRe/O9WMKvS+H/5CRPM5m7/PB4d62jl32jD1nL1nG9tkhe8eO2YQJ9ol9YV/Zt+RzcplcJd+vR3eSbeYp+wPJj5+oTqYn</latexit><latexit sha1_base64="bFz0n2P5KY7YuTtmLx+ZHinFcPo=">ACKnicdVDLihNBFK0eX2N8RV26uZgILkJT3Y6ZcTeDG5cjmJmBJDS3q29PiqlHU1UthCbf48ZfcTMLZXDrh1jJRFDRAwWHc+7h1j1lo6QPnF8lOzdu3rp9Z/du7979Bw8f9R8/OfG2dYImwirzkr0pKShSZB0VnjCHWp6LS8eLv2Tz+S89KaD2HZ0FzjuZG1FBiVPSPjgIQigXIQG6jcCjbhTBsC7kEByaymq1hNpZDcNZVxfZaFbZ4Ed1YWarYdEf8JS/zrM9DjzNx2/yg3Ek41f7Gc8hS/kGA7bFcdG/jHRajJBKPR+mvEmzDt0QpFq96s9dSguMBzmkZqUJOfd5tTV/AiKhXU1sVnAmzU3xMdau+XuoyTGsPC/+2txX950zbUB/NOmqYNZMT1orpVECyse4NKOhIh9lBJFE7Gv4JYoEMRe/O9WMKvS+H/5CRPM5m7/PB4d62jl32jD1nL1nG9tkhe8eO2YQJ9ol9YV/Zt+RzcplcJd+vR3eSbeYp+wPJj5+oTqYn</latexit>
slide-3
SLIDE 3

[1]: Léon Bottou. Curiously fast convergence of some stochastic gradient descent algorithms. 2009 [2]: Benjamin Recht and Christopher Ré. Beneath the valley of the noncommutative arithmetic-geometric mean inequality: conjectures, case-studies, and consequences. 2012

  • 1. Repeat K times

1. 2. Repeat n times

1. Sample fi uniformly at random from I 2. Remove fi from I 3.

Epoch

Known to be faster in practice! [1]

With v/s Without replacement [2]

xt+1 = xt αrfi(xt)

<latexit sha1_base64="IQdA3d61/L0pOIO3di72zHy1Sco=">AC3icbVDLSgNBEOz1bXzFx83LYBAUMeyKoBch4MWjglEhCWvZNYMzs4uM71iWHL34q948aCIV3/Am3/jJPHgq6ChqOqmuyvKlLTk+x/eyOjY+MTk1HRpZnZufqG8uHRm09xwUepSs1FhFYoqUWdJClxkRmBSaTEeXR92PfPb4SxMtWn1M1EK8ErLWPJkZwUltduw4K2gt7BbUjbTVRZB1lTY6SQxaHcOpmWK74VX8A9pcEX6RSW4njSwA4DsvzXbK80Ro4gqtbQR+Rq0CDUmuRK/UzK3IkF/jlWg4qjERtlUMfumxdae0WZwaV5rYQP0+UWBibTeJXGeC1LG/vb74n9fIKd5vFVJnOQnNh4viXDFKWT8Y1pZGcFJdR5Ab6W5lvIMGObn4Si6E4PfLf8nZTjXwq8FJUKntwhBTsAprsAEB7ENjuAY6sDhDh7gCZ69e+/Re/Feh60j3tfMvyA9/YJlpibiA=</latexit><latexit sha1_base64="sLutUCUuF7BoHlRg9Sm1xJdfmL8=">AC3icbVDJSgNBEO1xjXGLy81LYxAiYpgRQS9CwIvHCGaBJAw1nZ6ksadn6K6RhCF3L/6KFw+KePUHvPk3dpaD24OCx3tVNULEikMu6nMze/sLi0nFvJr6tb2wWtrbrJk414zUWy1g3AzBcCsVrKFDyZqI5RIHkjeD2cuw37rg2IlY3OEx4J4KeEqFgFbyC/sDP8Mjb3Qx8PG4DTLpA20rCTQ0Bclqx76haJbdiegf4k3I8XKbjhB1S98tLsxSyOukEkwpuW5CXYy0CiY5KN8OzU8AXYLPd6yVEHETSeb/DKiB1bp0jDWthTSifp9IoPImGEU2M4IsG9+e2PxP6+VYnjeyYRKUuSKTReFqaQY03EwtCs0ZyiHlgDTwt5KWR80MLTx5W0I3u+X/5L6Sdlzy961V6yckilyZI/skxLxyBmpkCtSJTXCyD15JM/kxXlwnpxX523aOufMZnbIDzjvXzeSnMA=</latexit><latexit sha1_base64="sLutUCUuF7BoHlRg9Sm1xJdfmL8=">AC3icbVDJSgNBEO1xjXGLy81LYxAiYpgRQS9CwIvHCGaBJAw1nZ6ksadn6K6RhCF3L/6KFw+KePUHvPk3dpaD24OCx3tVNULEikMu6nMze/sLi0nFvJr6tb2wWtrbrJk414zUWy1g3AzBcCsVrKFDyZqI5RIHkjeD2cuw37rg2IlY3OEx4J4KeEqFgFbyC/sDP8Mjb3Qx8PG4DTLpA20rCTQ0Bclqx76haJbdiegf4k3I8XKbjhB1S98tLsxSyOukEkwpuW5CXYy0CiY5KN8OzU8AXYLPd6yVEHETSeb/DKiB1bp0jDWthTSifp9IoPImGEU2M4IsG9+e2PxP6+VYnjeyYRKUuSKTReFqaQY03EwtCs0ZyiHlgDTwt5KWR80MLTx5W0I3u+X/5L6Sdlzy961V6yckilyZI/skxLxyBmpkCtSJTXCyD15JM/kxXlwnpxX523aOufMZnbIDzjvXzeSnMA=</latexit><latexit sha1_base64="DN0IQmTNHkJj2D6bCjn1SgvNIF0=">AC3icbVBNS8NAEN3U7/pV9ehlsQgVsSQi6EUQvHisYFVoQphsN+3SzSbsTqQl9O7Fv+LFgyJe/QPe/DduPw5+PRh4vDfDzLwok8Kg6346pZnZufmFxaXy8srq2nplY/PapLlmvMlSmerbCAyXQvEmCpT8NtMckjym6h3PvJv7rg2IlVXOMh4kEBHiVgwQCuFlZ1+WOC+Nzth3jg8y6QH0FkQah6Jm1b2wUnXr7hj0L/GmpEqmaISVD7+dsjzhCpkEY1qem2FQgEbBJB+W/dzwDFgPOrxlqYKEm6AY/zKku1Zp0zjVthTSsfp9oDEmES2c4EsGt+eyPxP6+VY3wSFEJlOXLFJoviXFJM6SgY2haM5QDS4BpYW+lrAsaGNr4yjYE7/fLf8n1Yd1z696lVz07msaxSLbJDqkRjxyTM3JBGqRJGLknj+SZvDgPzpPz6rxNWkvOdGaL/IDz/gWQDJoI</latexit>

I = {f1, . . . , fn}

<latexit sha1_base64="OIN0j95WIQeWuwniwD+ul0WXtA=">AB/3icbVC7SgNBFL0bXzG+okIam8EgWISwa6E2QsBGuwjmAdlmZ3MJkNmZ5eZWSGsKfwRERsLRWz9DTv/xsmj0MQDA4dz7uHeOUHCmdK2/W3lpZXVtfy64WNza3tneLuXlPFqS0QWIey3aAFeVM0IZmtN2IimOAk5bweBy7LfuqFQsFrd6mFAvwj3BQkawNpJfLF1fuFnoOxXkdmOtKij0hTsq+MWyXbUnQIvEmZFyrZQ8PgFA3S9+mTxJIyo04VipjmMn2suw1IxwOiq4qaIJgPcox1DBY6o8rLJ/SN0ZJQuCmNpntBov5OZDhSahgFZjLCuq/mvbH4n9dJdXjuZUwkqaCTBeFKUc6RuMyUJdJSjQfGoKJZOZWRPpYqJNZeMSnPkvL5LmSdWxq86NU6dwhR5OIBDOAYHzqAGV1CHBhC4h2d4hTfrwXqx3q2P6WjOmX24Q+szx9vEZaT</latexit><latexit sha1_base64="cMfWwFiPMji80SKaQDHDh/9t2/k=">AB/3icbVDLSgMxFM34rPU1KnTjJlgEF6VMXGg3QsGN7irYB3TKkEkzbWgmMyQZoYxd+BeuXLhxoYhbf8Od6MeYabvQ1gOBwzn3cG+OH3OmtON8WguLS8srq7m1/PrG5ta2vbPbUFEiCa2TiEey5WNFORO0rpnmtBVLikOf06Y/OM/85g2VikXiWg9j2glxT7CAEayN5NmFyzM3DTxUgm430qoEA0+4o7xnF52yMwacJ2hKitVC/HD/VWpefaHyZMkpEITjpVqIyfWnRLzQino7ybKBpjMsA92jZU4JCqTjq+fwQPjdKFQSTNExqO1d+JFIdKDUPfTIZY9Wsl4n/e1EB5VOykScaCrIZFGQcKgjmJUBu0xSovnQEwkM7dC0scSE20qy0pAs1+eJ43jMnLK6AoVqydghzYBwfgCBwCqrgAtRAHRBwCx7BM3ix7qwn69V6m4wuWNPMHvgD6/0H+l2Yeg=</latexit><latexit sha1_base64="cMfWwFiPMji80SKaQDHDh/9t2/k=">AB/3icbVDLSgMxFM34rPU1KnTjJlgEF6VMXGg3QsGN7irYB3TKkEkzbWgmMyQZoYxd+BeuXLhxoYhbf8Od6MeYabvQ1gOBwzn3cG+OH3OmtON8WguLS8srq7m1/PrG5ta2vbPbUFEiCa2TiEey5WNFORO0rpnmtBVLikOf06Y/OM/85g2VikXiWg9j2glxT7CAEayN5NmFyzM3DTxUgm430qoEA0+4o7xnF52yMwacJ2hKitVC/HD/VWpefaHyZMkpEITjpVqIyfWnRLzQino7ybKBpjMsA92jZU4JCqTjq+fwQPjdKFQSTNExqO1d+JFIdKDUPfTIZY9Wsl4n/e1EB5VOykScaCrIZFGQcKgjmJUBu0xSovnQEwkM7dC0scSE20qy0pAs1+eJ43jMnLK6AoVqydghzYBwfgCBwCqrgAtRAHRBwCx7BM3ix7qwn69V6m4wuWNPMHvgD6/0H+l2Yeg=</latexit><latexit sha1_base64="w72V17vDzh58MLO0QBQfIScPsIw=">AB/3icbVDLSsNAFJ34rPUVFdy4GSyCi1ISF+pGKLjRXQX7gCaEyXTSDp1MwsyNUGIX/obF4q49Tfc+TdO2y09cDA4Zx7uHdOmAquwXG+raXldW19dJGeXNre2fX3tv6SRTlDVpIhLVCYlmgkvWBA6CdVLFSBwK1g6H1xO/cCU5om8h1HK/Jj0JY84JWCkwD68vfLyKHCr2OsloKs4CqQ3Lgd2xak5U+BF4hakgo0AvL5GkWMwlUEK27rpOCnxMFnAo2LnuZimhQ9JnXUMliZn28+n9Y3xilB6OEmWeBDxVfydyEms9ikMzGRMY6HlvIv7ndTOILv2cyzQDJulsUZQJDAmelIF7XDEKYmQIoYqbWzEdEUomMomJbjzX14krbOa69TcO7dSPy/qKEjdIxOkYsuUB3doAZqIoe0TN6RW/Wk/VivVsfs9Elq8gcoD+wPn8AtsSUjw=</latexit>

SGD without replacement (SGDo)

slide-4
SLIDE 4

Why should SGDo be faster?

  • Example:
  • SGDo: Both functions seen in epoch, iterates stay close to 0.
  • SGD: With probability 1/2, f1 missed or f2 missed.
  • SGDo : Every function is seen once in n iterations.
  • SGD : In n iterations, n/e functions missed.

Variance over an epoch is reduced for SGDo!

Let f1(x) = (x + 1)2, f2(x) = (x − 1)2. Start at x = 0.

<latexit sha1_base64="wh7qNt7IHaW/WEg1mkEojUZq/MQ=">ACIHicbZDLSgMxFIYzXmu9V26CbZCizrMDELdFApuXLioaC/Q1pJM21o5kJyRlpKH8WNr+LGhSK606cxvSy09UDg5/vPSXJ+NxJcgWV9GUvLK6tr64mN5ObW9s5uam+/osJYUlamoQhlzSWKCR6wMnAQrBZJRnxXsKrbuxz71QcmFQ+DOxhErOmTsA9Tglo1ErlrxngjNeys/1cIds/sXP3TuZ0TJwpOZsQE98CkYCJbu4XrIzZSqUt05oUXhT2TKTRrEqt1GejHdLYZwFQZSq21YEzaG+lFPBRslGrFhEaI90WF3LgPhMNYeTBUf4WJM29kKpTwB4Qn9PDImv1MB3dadPoKvmvTH8z6vH4F0hzyIYmABnT7kxQJDiMdp4TaXjIYaEGo5PqvmHaJBR0pkdgj2/8qKoOKZtmfaNky6ez+JIoEN0hLIRnlURFeohMqIokf0jF7Rm/FkvBjvxse0dcmYzRygP2V8/wC1KJ0T</latexit><latexit sha1_base64="wh7qNt7IHaW/WEg1mkEojUZq/MQ=">ACIHicbZDLSgMxFIYzXmu9V26CbZCizrMDELdFApuXLioaC/Q1pJM21o5kJyRlpKH8WNr+LGhSK606cxvSy09UDg5/vPSXJ+NxJcgWV9GUvLK6tr64mN5ObW9s5uam+/osJYUlamoQhlzSWKCR6wMnAQrBZJRnxXsKrbuxz71QcmFQ+DOxhErOmTsA9Tglo1ErlrxngjNeys/1cIds/sXP3TuZ0TJwpOZsQE98CkYCJbu4XrIzZSqUt05oUXhT2TKTRrEqt1GejHdLYZwFQZSq21YEzaG+lFPBRslGrFhEaI90WF3LgPhMNYeTBUf4WJM29kKpTwB4Qn9PDImv1MB3dadPoKvmvTH8z6vH4F0hzyIYmABnT7kxQJDiMdp4TaXjIYaEGo5PqvmHaJBR0pkdgj2/8qKoOKZtmfaNky6ez+JIoEN0hLIRnlURFeohMqIokf0jF7Rm/FkvBjvxse0dcmYzRygP2V8/wC1KJ0T</latexit><latexit sha1_base64="wh7qNt7IHaW/WEg1mkEojUZq/MQ=">ACIHicbZDLSgMxFIYzXmu9V26CbZCizrMDELdFApuXLioaC/Q1pJM21o5kJyRlpKH8WNr+LGhSK606cxvSy09UDg5/vPSXJ+NxJcgWV9GUvLK6tr64mN5ObW9s5uam+/osJYUlamoQhlzSWKCR6wMnAQrBZJRnxXsKrbuxz71QcmFQ+DOxhErOmTsA9Tglo1ErlrxngjNeys/1cIds/sXP3TuZ0TJwpOZsQE98CkYCJbu4XrIzZSqUt05oUXhT2TKTRrEqt1GejHdLYZwFQZSq21YEzaG+lFPBRslGrFhEaI90WF3LgPhMNYeTBUf4WJM29kKpTwB4Qn9PDImv1MB3dadPoKvmvTH8z6vH4F0hzyIYmABnT7kxQJDiMdp4TaXjIYaEGo5PqvmHaJBR0pkdgj2/8qKoOKZtmfaNky6ez+JIoEN0hLIRnlURFeohMqIokf0jF7Rm/FkvBjvxse0dcmYzRygP2V8/wC1KJ0T</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="KXevPcd9apzCl8I+x8SVakV8cLQ=">ACFXicbZDNTgIxFIXv+IuIim7dNIoJRJ3MsNGNiYkbFy4wipDASDqlg42dn7R3DIT4KG58FTcuNEZ3vo0dYKHgSZqcfPf2tvf4iRQaHefbmptfWFxazq3kVwtr6xvFzcKNjlPFeJ3FMlZNn2ouRcTrKFDyZqI4DX3JG/79WVZvPHClRxd4yDhXkh7kQgEo2hQp3h0wZGUgo5b7ldOyv19t3JbLR1kpDomhyNikyukCgk1zf0Tp2R3iruO7YxEZo07MbswUa1T/Gp3Y5aGPEImqdYt10nQG5qhgkn+mG+nmieU3dMebxkb0ZBrbzha8JHsGdIlQazMiZCM6O8bQxpqPQh90xlSvNPTtQz+V2ulGBx7QxElKfKIjR8KUkwJlapCsUZygHxlCmhPkrYXdUYm07wJwZ1edbcVG3Xsd1LB3KwDTtQBheO4BTOoQZ1YPAEL/AG79az9Wp9jOasya5bcEfWZ8/aybmA=</latexit><latexit sha1_base64="KXevPcd9apzCl8I+x8SVakV8cLQ=">ACFXicbZDNTgIxFIXv+IuIim7dNIoJRJ3MsNGNiYkbFy4wipDASDqlg42dn7R3DIT4KG58FTcuNEZ3vo0dYKHgSZqcfPf2tvf4iRQaHefbmptfWFxazq3kVwtr6xvFzcKNjlPFeJ3FMlZNn2ouRcTrKFDyZqI4DX3JG/79WVZvPHClRxd4yDhXkh7kQgEo2hQp3h0wZGUgo5b7ldOyv19t3JbLR1kpDomhyNikyukCgk1zf0Tp2R3iruO7YxEZo07MbswUa1T/Gp3Y5aGPEImqdYt10nQG5qhgkn+mG+nmieU3dMebxkb0ZBrbzha8JHsGdIlQazMiZCM6O8bQxpqPQh90xlSvNPTtQz+V2ulGBx7QxElKfKIjR8KUkwJlapCsUZygHxlCmhPkrYXdUYm07wJwZ1edbcVG3Xsd1LB3KwDTtQBheO4BTOoQZ1YPAEL/AG79az9Wp9jOasya5bcEfWZ8/aybmA=</latexit><latexit sha1_base64="+hMwlBr/P54Woq1GBpLm2uSIhw=">ACIHicbZDLSgMxFIYzXmu9jbp0E2yFnWYKULdFApuXLioaC/Q1pJM21o5kJyRlpKH8WNr+LGhSK606cxvSy09UDg5/vPSXJ+NxJcgW1/GUvLK6tr64mN5ObW9s6ubdfUWEsKSvTUISy5hLFBA9YGTgIVoskI74rWNXtXY796gOTiofBHQwi1vRJ+AepwQ0apn5awY47bWcTD9byPRPnOx9Ln06JrkpOZsQC98CkYCJbu4X7LTVMlO2ZU8KLwpnJlJoVqW+dlohzT2WQBUEKXqjh1Bc6gv5VSwUbIRKxYR2iMdVtcyID5TzeFkwRE+1qSNvVDqEwCe0N8TQ+IrNfBd3ekT6Kp5bwz/8+oxeBfNIQ+iGFhApw95scAQ4nFauM0loyAGWhAquf4rpl0iCQWdaVKH4MyvCgqOcuxLefGThXPZ3Ek0CE6QhnkoDwqoitUQmVE0SN6Rq/ozXgyXox342PaumTMZg7QnzK+fwC0iJ0R</latexit><latexit sha1_base64="wh7qNt7IHaW/WEg1mkEojUZq/MQ=">ACIHicbZDLSgMxFIYzXmu9V26CbZCizrMDELdFApuXLioaC/Q1pJM21o5kJyRlpKH8WNr+LGhSK606cxvSy09UDg5/vPSXJ+NxJcgWV9GUvLK6tr64mN5ObW9s5uam+/osJYUlamoQhlzSWKCR6wMnAQrBZJRnxXsKrbuxz71QcmFQ+DOxhErOmTsA9Tglo1ErlrxngjNeys/1cIds/sXP3TuZ0TJwpOZsQE98CkYCJbu4XrIzZSqUt05oUXhT2TKTRrEqt1GejHdLYZwFQZSq21YEzaG+lFPBRslGrFhEaI90WF3LgPhMNYeTBUf4WJM29kKpTwB4Qn9PDImv1MB3dadPoKvmvTH8z6vH4F0hzyIYmABnT7kxQJDiMdp4TaXjIYaEGo5PqvmHaJBR0pkdgj2/8qKoOKZtmfaNky6ez+JIoEN0hLIRnlURFeohMqIokf0jF7Rm/FkvBjvxse0dcmYzRygP2V8/wC1KJ0T</latexit><latexit sha1_base64="wh7qNt7IHaW/WEg1mkEojUZq/MQ=">ACIHicbZDLSgMxFIYzXmu9V26CbZCizrMDELdFApuXLioaC/Q1pJM21o5kJyRlpKH8WNr+LGhSK606cxvSy09UDg5/vPSXJ+NxJcgWV9GUvLK6tr64mN5ObW9s5uam+/osJYUlamoQhlzSWKCR6wMnAQrBZJRnxXsKrbuxz71QcmFQ+DOxhErOmTsA9Tglo1ErlrxngjNeys/1cIds/sXP3TuZ0TJwpOZsQE98CkYCJbu4XrIzZSqUt05oUXhT2TKTRrEqt1GejHdLYZwFQZSq21YEzaG+lFPBRslGrFhEaI90WF3LgPhMNYeTBUf4WJM29kKpTwB4Qn9PDImv1MB3dadPoKvmvTH8z6vH4F0hzyIYmABnT7kxQJDiMdp4TaXjIYaEGo5PqvmHaJBR0pkdgj2/8qKoOKZtmfaNky6ez+JIoEN0hLIRnlURFeohMqIokf0jF7Rm/FkvBjvxse0dcmYzRygP2V8/wC1KJ0T</latexit><latexit sha1_base64="wh7qNt7IHaW/WEg1mkEojUZq/MQ=">ACIHicbZDLSgMxFIYzXmu9V26CbZCizrMDELdFApuXLioaC/Q1pJM21o5kJyRlpKH8WNr+LGhSK606cxvSy09UDg5/vPSXJ+NxJcgWV9GUvLK6tr64mN5ObW9s5uam+/osJYUlamoQhlzSWKCR6wMnAQrBZJRnxXsKrbuxz71QcmFQ+DOxhErOmTsA9Tglo1ErlrxngjNeys/1cIds/sXP3TuZ0TJwpOZsQE98CkYCJbu4XrIzZSqUt05oUXhT2TKTRrEqt1GejHdLYZwFQZSq21YEzaG+lFPBRslGrFhEaI90WF3LgPhMNYeTBUf4WJM29kKpTwB4Qn9PDImv1MB3dadPoKvmvTH8z6vH4F0hzyIYmABnT7kxQJDiMdp4TaXjIYaEGo5PqvmHaJBR0pkdgj2/8qKoOKZtmfaNky6ez+JIoEN0hLIRnlURFeohMqIokf0jF7Rm/FkvBjvxse0dcmYzRygP2V8/wC1KJ0T</latexit><latexit sha1_base64="wh7qNt7IHaW/WEg1mkEojUZq/MQ=">ACIHicbZDLSgMxFIYzXmu9V26CbZCizrMDELdFApuXLioaC/Q1pJM21o5kJyRlpKH8WNr+LGhSK606cxvSy09UDg5/vPSXJ+NxJcgWV9GUvLK6tr64mN5ObW9s5uam+/osJYUlamoQhlzSWKCR6wMnAQrBZJRnxXsKrbuxz71QcmFQ+DOxhErOmTsA9Tglo1ErlrxngjNeys/1cIds/sXP3TuZ0TJwpOZsQE98CkYCJbu4XrIzZSqUt05oUXhT2TKTRrEqt1GejHdLYZwFQZSq21YEzaG+lFPBRslGrFhEaI90WF3LgPhMNYeTBUf4WJM29kKpTwB4Qn9PDImv1MB3dadPoKvmvTH8z6vH4F0hzyIYmABnT7kxQJDiMdp4TaXjIYaEGo5PqvmHaJBR0pkdgj2/8qKoOKZtmfaNky6ez+JIoEN0hLIRnlURFeohMqIokf0jF7Rm/FkvBjvxse0dcmYzRygP2V8/wC1KJ0T</latexit><latexit sha1_base64="wh7qNt7IHaW/WEg1mkEojUZq/MQ=">ACIHicbZDLSgMxFIYzXmu9V26CbZCizrMDELdFApuXLioaC/Q1pJM21o5kJyRlpKH8WNr+LGhSK606cxvSy09UDg5/vPSXJ+NxJcgWV9GUvLK6tr64mN5ObW9s5uam+/osJYUlamoQhlzSWKCR6wMnAQrBZJRnxXsKrbuxz71QcmFQ+DOxhErOmTsA9Tglo1ErlrxngjNeys/1cIds/sXP3TuZ0TJwpOZsQE98CkYCJbu4XrIzZSqUt05oUXhT2TKTRrEqt1GejHdLYZwFQZSq21YEzaG+lFPBRslGrFhEaI90WF3LgPhMNYeTBUf4WJM29kKpTwB4Qn9PDImv1MB3dadPoKvmvTH8z6vH4F0hzyIYmABnT7kxQJDiMdp4TaXjIYaEGo5PqvmHaJBR0pkdgj2/8qKoOKZtmfaNky6ez+JIoEN0hLIRnlURFeohMqIokf0jF7Rm/FkvBjvxse0dcmYzRygP2V8/wC1KJ0T</latexit><latexit sha1_base64="wh7qNt7IHaW/WEg1mkEojUZq/MQ=">ACIHicbZDLSgMxFIYzXmu9V26CbZCizrMDELdFApuXLioaC/Q1pJM21o5kJyRlpKH8WNr+LGhSK606cxvSy09UDg5/vPSXJ+NxJcgWV9GUvLK6tr64mN5ObW9s5uam+/osJYUlamoQhlzSWKCR6wMnAQrBZJRnxXsKrbuxz71QcmFQ+DOxhErOmTsA9Tglo1ErlrxngjNeys/1cIds/sXP3TuZ0TJwpOZsQE98CkYCJbu4XrIzZSqUt05oUXhT2TKTRrEqt1GejHdLYZwFQZSq21YEzaG+lFPBRslGrFhEaI90WF3LgPhMNYeTBUf4WJM29kKpTwB4Qn9PDImv1MB3dadPoKvmvTH8z6vH4F0hzyIYmABnT7kxQJDiMdp4TaXjIYaEGo5PqvmHaJBR0pkdgj2/8qKoOKZtmfaNky6ez+JIoEN0hLIRnlURFeohMqIokf0jF7Rm/FkvBjvxse0dcmYzRygP2V8/wC1KJ0T</latexit>
slide-5
SLIDE 5

SGDo – Theoretically elusive

  • Until recently, SGDo eluded theoretical analysis
  • Why?
  • Can SGDo (provably) do better?

SGD: Easy because E[rfi(xt)] = rF(xt)

<latexit sha1_base64="VpAePuJkJQCH5BuUrfKPVO+S3YU=">ACJnicbVDLSgMxFM34tr6qLt0Eq6CbMlMERAEn0tF+4B2G7STBvMZIYkI5ahX+PGX3HjQhFx56eYjrPwdSBwcs693HsPSQTXxnXfnbHxicmp6ZnZ0tz8wuJSeXmloeNUVansYhVi4BmgktWN9wI1koUg4gI1iQ3RyO/ecuU5rG8NoOE+RH0JA85BWOloHxwdXa8j09ADzBhFLN8EYnAtMnJDsZtjsSiAcBnzrLjDb/kEhnObfjaBcatuDvyXeAWpoAIXQfm5041pGjFpqACt256bGD8DZTgVbFjq2AUSoDfQY21LJURM+1l+5hBvWqWLw1jZJw3O1e8dGURaDyJiK0cX6N/eSPzPa6cm3PMzLpPUMEm/BoWpwCbGo8xwlytGjRhYAlRxuyumfVBAjU2ZEPwfp/8lzRqVc+tepe1yuFOEcMWkPraAt5aBcdonN0geqIonv0iJ7Ri/PgPDmvztX6ZhT9KyiH3A+PgFNwqPD</latexit><latexit sha1_base64="VpAePuJkJQCH5BuUrfKPVO+S3YU=">ACJnicbVDLSgMxFM34tr6qLt0Eq6CbMlMERAEn0tF+4B2G7STBvMZIYkI5ahX+PGX3HjQhFx56eYjrPwdSBwcs693HsPSQTXxnXfnbHxicmp6ZnZ0tz8wuJSeXmloeNUVansYhVi4BmgktWN9wI1koUg4gI1iQ3RyO/ecuU5rG8NoOE+RH0JA85BWOloHxwdXa8j09ADzBhFLN8EYnAtMnJDsZtjsSiAcBnzrLjDb/kEhnObfjaBcatuDvyXeAWpoAIXQfm5041pGjFpqACt256bGD8DZTgVbFjq2AUSoDfQY21LJURM+1l+5hBvWqWLw1jZJw3O1e8dGURaDyJiK0cX6N/eSPzPa6cm3PMzLpPUMEm/BoWpwCbGo8xwlytGjRhYAlRxuyumfVBAjU2ZEPwfp/8lzRqVc+tepe1yuFOEcMWkPraAt5aBcdonN0geqIonv0iJ7Ri/PgPDmvztX6ZhT9KyiH3A+PgFNwqPD</latexit><latexit sha1_base64="VpAePuJkJQCH5BuUrfKPVO+S3YU=">ACJnicbVDLSgMxFM34tr6qLt0Eq6CbMlMERAEn0tF+4B2G7STBvMZIYkI5ahX+PGX3HjQhFx56eYjrPwdSBwcs693HsPSQTXxnXfnbHxicmp6ZnZ0tz8wuJSeXmloeNUVansYhVi4BmgktWN9wI1koUg4gI1iQ3RyO/ecuU5rG8NoOE+RH0JA85BWOloHxwdXa8j09ADzBhFLN8EYnAtMnJDsZtjsSiAcBnzrLjDb/kEhnObfjaBcatuDvyXeAWpoAIXQfm5041pGjFpqACt256bGD8DZTgVbFjq2AUSoDfQY21LJURM+1l+5hBvWqWLw1jZJw3O1e8dGURaDyJiK0cX6N/eSPzPa6cm3PMzLpPUMEm/BoWpwCbGo8xwlytGjRhYAlRxuyumfVBAjU2ZEPwfp/8lzRqVc+tepe1yuFOEcMWkPraAt5aBcdonN0geqIonv0iJ7Ri/PgPDmvztX6ZhT9KyiH3A+PgFNwqPD</latexit><latexit sha1_base64="VpAePuJkJQCH5BuUrfKPVO+S3YU=">ACJnicbVDLSgMxFM34tr6qLt0Eq6CbMlMERAEn0tF+4B2G7STBvMZIYkI5ahX+PGX3HjQhFx56eYjrPwdSBwcs693HsPSQTXxnXfnbHxicmp6ZnZ0tz8wuJSeXmloeNUVansYhVi4BmgktWN9wI1koUg4gI1iQ3RyO/ecuU5rG8NoOE+RH0JA85BWOloHxwdXa8j09ADzBhFLN8EYnAtMnJDsZtjsSiAcBnzrLjDb/kEhnObfjaBcatuDvyXeAWpoAIXQfm5041pGjFpqACt256bGD8DZTgVbFjq2AUSoDfQY21LJURM+1l+5hBvWqWLw1jZJw3O1e8dGURaDyJiK0cX6N/eSPzPa6cm3PMzLpPUMEm/BoWpwCbGo8xwlytGjRhYAlRxuyumfVBAjU2ZEPwfp/8lzRqVc+tepe1yuFOEcMWkPraAt5aBcdonN0geqIonv0iJ7Ri/PgPDmvztX6ZhT9KyiH3A+PgFNwqPD</latexit>

SGDo: Difficult because E[rfi(xt)] 6= rF(xt)

<latexit sha1_base64="J1i6DezY75sNWnsHnq9oDH70tFE=">ACL3icbZBNSxBEIZ71BizRrPRYy6Nq6CXZUaEhJwEv3JUdFXYGYbq3hpt7OmZdNcEl2H/kRf/ipcQDCHX/Iv0rnPw64WGl6eq6KpXlFo5CsNfwdT0zJvZt3PvWvPvFxY/tD8unbqishJ7stCFPRfgUCuDPVKk8by0CLnQeCaudsb1sx9onSrMCQ1LTHK4MCpTEsijtL1/fLBbfOW7KvOs0sQFSqgc8tU4B7oUot4b9WMDQgPUrV+ndJGEhv83rD9CVlN252wG07EX5qoMR3W6DBt38WDQlY5GpIanOtHYUlJDZaU1DhqxX6HEuQVXGDfWwM5uqSe3Dvia54MeFZY/wzxCX08UPu3DAXvnN8hHteG8PXav2Ksi9JrUxZERr58FWaU4FH4fHB8qiJD30BqRVflcuL8GCJB9xy4cQPT/5pTnd7EZhNzra7GxvNXHMsU9sha2ziH1m2+wbO2Q9JtkNu2P37HdwG/wM/gR/H1qngmZmT1R8O8/WuOn9w=</latexit><latexit sha1_base64="J1i6DezY75sNWnsHnq9oDH70tFE=">ACL3icbZBNSxBEIZ71BizRrPRYy6Nq6CXZUaEhJwEv3JUdFXYGYbq3hpt7OmZdNcEl2H/kRf/ipcQDCHX/Iv0rnPw64WGl6eq6KpXlFo5CsNfwdT0zJvZt3PvWvPvFxY/tD8unbqishJ7stCFPRfgUCuDPVKk8by0CLnQeCaudsb1sx9onSrMCQ1LTHK4MCpTEsijtL1/fLBbfOW7KvOs0sQFSqgc8tU4B7oUot4b9WMDQgPUrV+ndJGEhv83rD9CVlN252wG07EX5qoMR3W6DBt38WDQlY5GpIanOtHYUlJDZaU1DhqxX6HEuQVXGDfWwM5uqSe3Dvia54MeFZY/wzxCX08UPu3DAXvnN8hHteG8PXav2Ksi9JrUxZERr58FWaU4FH4fHB8qiJD30BqRVflcuL8GCJB9xy4cQPT/5pTnd7EZhNzra7GxvNXHMsU9sha2ziH1m2+wbO2Q9JtkNu2P37HdwG/wM/gR/H1qngmZmT1R8O8/WuOn9w=</latexit><latexit sha1_base64="J1i6DezY75sNWnsHnq9oDH70tFE=">ACL3icbZBNSxBEIZ71BizRrPRYy6Nq6CXZUaEhJwEv3JUdFXYGYbq3hpt7OmZdNcEl2H/kRf/ipcQDCHX/Iv0rnPw64WGl6eq6KpXlFo5CsNfwdT0zJvZt3PvWvPvFxY/tD8unbqishJ7stCFPRfgUCuDPVKk8by0CLnQeCaudsb1sx9onSrMCQ1LTHK4MCpTEsijtL1/fLBbfOW7KvOs0sQFSqgc8tU4B7oUot4b9WMDQgPUrV+ndJGEhv83rD9CVlN252wG07EX5qoMR3W6DBt38WDQlY5GpIanOtHYUlJDZaU1DhqxX6HEuQVXGDfWwM5uqSe3Dvia54MeFZY/wzxCX08UPu3DAXvnN8hHteG8PXav2Ksi9JrUxZERr58FWaU4FH4fHB8qiJD30BqRVflcuL8GCJB9xy4cQPT/5pTnd7EZhNzra7GxvNXHMsU9sha2ziH1m2+wbO2Q9JtkNu2P37HdwG/wM/gR/H1qngmZmT1R8O8/WuOn9w=</latexit><latexit sha1_base64="J1i6DezY75sNWnsHnq9oDH70tFE=">ACL3icbZBNSxBEIZ71BizRrPRYy6Nq6CXZUaEhJwEv3JUdFXYGYbq3hpt7OmZdNcEl2H/kRf/ipcQDCHX/Iv0rnPw64WGl6eq6KpXlFo5CsNfwdT0zJvZt3PvWvPvFxY/tD8unbqishJ7stCFPRfgUCuDPVKk8by0CLnQeCaudsb1sx9onSrMCQ1LTHK4MCpTEsijtL1/fLBbfOW7KvOs0sQFSqgc8tU4B7oUot4b9WMDQgPUrV+ndJGEhv83rD9CVlN252wG07EX5qoMR3W6DBt38WDQlY5GpIanOtHYUlJDZaU1DhqxX6HEuQVXGDfWwM5uqSe3Dvia54MeFZY/wzxCX08UPu3DAXvnN8hHteG8PXav2Ksi9JrUxZERr58FWaU4FH4fHB8qiJD30BqRVflcuL8GCJB9xy4cQPT/5pTnd7EZhNzra7GxvNXHMsU9sha2ziH1m2+wbO2Q9JtkNu2P37HdwG/wM/gR/H1qngmZmT1R8O8/WuOn9w=</latexit>

Error metric: E[kxT x∗k2]

<latexit sha1_base64="Nj/lTRxwjIt40CJcHNxIPI9ygxI=">ACEnicbVC7TgJBFJ3F+Jr1dJmIpioiWSXRmJFYkgsMeGVsAuZHWZhwuwjM7MGsvANv6KjYXG2FrZ+TfOwhYKnurknHtz1OyKiQhvGtZdbWNza3stu5nd29/QP98Kgpgohj0sABC3jbQYIw6pOGpJKRdsgJ8hxGWs7oNvFbD4QLGvh1OQmJ7aGBT12KkVRST7+och5w6BHJKb6BctDcug4cXWsabjXv1q3L20pt2SXejpeaNozAFXiZmSPEhR6+lfVj/AkUd8iRkSomMaobRjxCXFjMxyViRIiPAIDUhHUR95RNjx/KUZPFNKH7oqmhv4Es7V3xsx8oSYeI6aTBKLZS8R/M6kXTLdkz9MJLEx4tDbsSgDGDSD+xTrBkE0UQ5lRlhXiIOMJStZhTJZjL6+SZqloGkXzvpSvlNM6suAEnIJzYIJrUAF3oAYaAINH8AxewZv2pL1o79rHYjSjpTvH4A+0zx/U1Jzg</latexit><latexit sha1_base64="Nj/lTRxwjIt40CJcHNxIPI9ygxI=">ACEnicbVC7TgJBFJ3F+Jr1dJmIpioiWSXRmJFYkgsMeGVsAuZHWZhwuwjM7MGsvANv6KjYXG2FrZ+TfOwhYKnurknHtz1OyKiQhvGtZdbWNza3stu5nd29/QP98Kgpgohj0sABC3jbQYIw6pOGpJKRdsgJ8hxGWs7oNvFbD4QLGvh1OQmJ7aGBT12KkVRST7+och5w6BHJKb6BctDcug4cXWsabjXv1q3L20pt2SXejpeaNozAFXiZmSPEhR6+lfVj/AkUd8iRkSomMaobRjxCXFjMxyViRIiPAIDUhHUR95RNjx/KUZPFNKH7oqmhv4Es7V3xsx8oSYeI6aTBKLZS8R/M6kXTLdkz9MJLEx4tDbsSgDGDSD+xTrBkE0UQ5lRlhXiIOMJStZhTJZjL6+SZqloGkXzvpSvlNM6suAEnIJzYIJrUAF3oAYaAINH8AxewZv2pL1o79rHYjSjpTvH4A+0zx/U1Jzg</latexit><latexit sha1_base64="Nj/lTRxwjIt40CJcHNxIPI9ygxI=">ACEnicbVC7TgJBFJ3F+Jr1dJmIpioiWSXRmJFYkgsMeGVsAuZHWZhwuwjM7MGsvANv6KjYXG2FrZ+TfOwhYKnurknHtz1OyKiQhvGtZdbWNza3stu5nd29/QP98Kgpgohj0sABC3jbQYIw6pOGpJKRdsgJ8hxGWs7oNvFbD4QLGvh1OQmJ7aGBT12KkVRST7+och5w6BHJKb6BctDcug4cXWsabjXv1q3L20pt2SXejpeaNozAFXiZmSPEhR6+lfVj/AkUd8iRkSomMaobRjxCXFjMxyViRIiPAIDUhHUR95RNjx/KUZPFNKH7oqmhv4Es7V3xsx8oSYeI6aTBKLZS8R/M6kXTLdkz9MJLEx4tDbsSgDGDSD+xTrBkE0UQ5lRlhXiIOMJStZhTJZjL6+SZqloGkXzvpSvlNM6suAEnIJzYIJrUAF3oAYaAINH8AxewZv2pL1o79rHYjSjpTvH4A+0zx/U1Jzg</latexit><latexit sha1_base64="Nj/lTRxwjIt40CJcHNxIPI9ygxI=">ACEnicbVC7TgJBFJ3F+Jr1dJmIpioiWSXRmJFYkgsMeGVsAuZHWZhwuwjM7MGsvANv6KjYXG2FrZ+TfOwhYKnurknHtz1OyKiQhvGtZdbWNza3stu5nd29/QP98Kgpgohj0sABC3jbQYIw6pOGpJKRdsgJ8hxGWs7oNvFbD4QLGvh1OQmJ7aGBT12KkVRST7+och5w6BHJKb6BctDcug4cXWsabjXv1q3L20pt2SXejpeaNozAFXiZmSPEhR6+lfVj/AkUd8iRkSomMaobRjxCXFjMxyViRIiPAIDUhHUR95RNjx/KUZPFNKH7oqmhv4Es7V3xsx8oSYeI6aTBKLZS8R/M6kXTLdkz9MJLEx4tDbsSgDGDSD+xTrBkE0UQ5lRlhXiIOMJStZhTJZjL6+SZqloGkXzvpSvlNM6suAEnIJzYIJrUAF3oAYaAINH8AxewZv2pL1o79rHYjSjpTvH4A+0zx/U1Jzg</latexit>

SGD error = O 1

T

  • <latexit sha1_base64="ZfrSnC2Y4QdkDnvbaewdRclM0U=">ACD3icbVC7SgNBFJ2NrxhfUubwajEJuzamEYIKGhnRKNCdgmzk7vJ4OyDmbtCWPYPbPwVGwtFbG3t/BsnMYUmnupwzr3c4+fSKHRtr+swszs3PxCcbG0tLyulZe37jWcao4tHgsY3XrMw1SRNBCgRJuEwUs9CXc+HfHQ/mHpQWcXSFgwS8kPUiEQjO0Eid8t7l6QkFpWJFd47OXQkBVt1AMZ45eXaVu0r0+ri/0ylX7Jo9Ap0mzphUyBjNTvnT7cY8DSFCLpnWbcdO0MuYQsEl5CU31ZAwfsd60DY0YiFoLxv9k9Ndo3RpYDIFcYR0pP7eyFio9SD0zWTIsK8nvaH4n9dOMah7mYiSFCHiP4eCVFKM6bAc2hUKOMqBIYwrYbJS3memDTQVlkwJzuTL0+T6oObYNefioNKoj+soki2yTarEIYekQc5Ik7QIJw/kibyQV+vRerberPef0YI13tkf2B9fAMwLpt2</latexit><latexit sha1_base64="ZfrSnC2Y4QdkDnvbaewdRclM0U=">ACD3icbVC7SgNBFJ2NrxhfUubwajEJuzamEYIKGhnRKNCdgmzk7vJ4OyDmbtCWPYPbPwVGwtFbG3t/BsnMYUmnupwzr3c4+fSKHRtr+swszs3PxCcbG0tLyulZe37jWcao4tHgsY3XrMw1SRNBCgRJuEwUs9CXc+HfHQ/mHpQWcXSFgwS8kPUiEQjO0Eid8t7l6QkFpWJFd47OXQkBVt1AMZ45eXaVu0r0+ri/0ylX7Jo9Ap0mzphUyBjNTvnT7cY8DSFCLpnWbcdO0MuYQsEl5CU31ZAwfsd60DY0YiFoLxv9k9Ndo3RpYDIFcYR0pP7eyFio9SD0zWTIsK8nvaH4n9dOMah7mYiSFCHiP4eCVFKM6bAc2hUKOMqBIYwrYbJS3memDTQVlkwJzuTL0+T6oObYNefioNKoj+soki2yTarEIYekQc5Ik7QIJw/kibyQV+vRerberPef0YI13tkf2B9fAMwLpt2</latexit><latexit sha1_base64="ZfrSnC2Y4QdkDnvbaewdRclM0U=">ACD3icbVC7SgNBFJ2NrxhfUubwajEJuzamEYIKGhnRKNCdgmzk7vJ4OyDmbtCWPYPbPwVGwtFbG3t/BsnMYUmnupwzr3c4+fSKHRtr+swszs3PxCcbG0tLyulZe37jWcao4tHgsY3XrMw1SRNBCgRJuEwUs9CXc+HfHQ/mHpQWcXSFgwS8kPUiEQjO0Eid8t7l6QkFpWJFd47OXQkBVt1AMZ45eXaVu0r0+ri/0ylX7Jo9Ap0mzphUyBjNTvnT7cY8DSFCLpnWbcdO0MuYQsEl5CU31ZAwfsd60DY0YiFoLxv9k9Ndo3RpYDIFcYR0pP7eyFio9SD0zWTIsK8nvaH4n9dOMah7mYiSFCHiP4eCVFKM6bAc2hUKOMqBIYwrYbJS3memDTQVlkwJzuTL0+T6oObYNefioNKoj+soki2yTarEIYekQc5Ik7QIJw/kibyQV+vRerberPef0YI13tkf2B9fAMwLpt2</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="z2A32rOjl4umhmagCuHNMv7ucs=">ACBHicbZC7SgNBFIbPeo3xFm1tBhNFm7Bro40gKGhnxESFbAizk7PJ4OyFmbNCWPIGNr6KjYUiPoOdb+PkUmj0r37+/wxzhekShpy3S9nZnZufmGxsFRcXldWy9trNyYJNMCGyJRib4LuElY2yQJIV3qUYeBQpvg/vTYX/7gNrIJK5TP8VWxLuxDKXgZKN2af6/Iyh1olmleNLX2FIe36ouci9QV4f+Fp2e7RfaZfKbtUdif013sSUYaJau/TpdxKRiTUNyYpuem1Mq5JikUDop+ZjDl4p53sWltzCM0rXx0z4Dt2KTDQrtTmMTERunPFzmPjOlHgZ2MOPXMdDcM/+uaGYVHrVzGaUYi/FHYaYJWwIh3WkRkGqbw0XWtpdmehxS4MswqKF4E2f/NfcHFQ9t+pduVCALdiGPfDgE7gAmrQAGP8Ayv8OY8OS/O+xjXjDPhtgm/5Hx8A4Zbmg=</latexit><latexit sha1_base64="z2A32rOjl4umhmagCuHNMv7ucs=">ACBHicbZC7SgNBFIbPeo3xFm1tBhNFm7Bro40gKGhnxESFbAizk7PJ4OyFmbNCWPIGNr6KjYUiPoOdb+PkUmj0r37+/wxzhekShpy3S9nZnZufmGxsFRcXldWy9trNyYJNMCGyJRib4LuElY2yQJIV3qUYeBQpvg/vTYX/7gNrIJK5TP8VWxLuxDKXgZKN2af6/Iyh1olmleNLX2FIe36ouci9QV4f+Fp2e7RfaZfKbtUdif013sSUYaJau/TpdxKRiTUNyYpuem1Mq5JikUDop+ZjDl4p53sWltzCM0rXx0z4Dt2KTDQrtTmMTERunPFzmPjOlHgZ2MOPXMdDcM/+uaGYVHrVzGaUYi/FHYaYJWwIh3WkRkGqbw0XWtpdmehxS4MswqKF4E2f/NfcHFQ9t+pduVCALdiGPfDgE7gAmrQAGP8Ayv8OY8OS/O+xjXjDPhtgm/5Hx8A4Zbmg=</latexit><latexit sha1_base64="GfSFVCW46W7o5TdiHyrid1jY4ck=">ACD3icbVC7SgNBFJ31bXxFLW0GEyU2YdG0FQ0E7FvC7hNnJ3WRw9sHMXSEs+QMbf8XGQhFbWzv/xkncQhNPdTjnXu65x0+k0GjbX9bM7Nz8wuLScmFldW19o7i51dBxqjUeSxj1fKZBikiqKNACa1EAQt9CU3/7mzkN+9BaRFHNRwk4IWsF4lAcIZG6hT3by/OKSgVK1o+uXIlBFhxA8V45gyz2tBVotfHg3KnWLKr9h0mjg5KZEc153ip9uNeRpChFwyrduOnaCXMYWCSxgW3FRDwvgd60Hb0IiFoL1s/M+Q7hmlSwOTKYgjpGP190bGQq0HoW8mQ4Z9PemNxP+8dorBsZeJKEkRIv5zKEglxZiOyqFdoYCjHBjCuBImK+V9ZtpAU2HBlOBMvjxNGodVx646N3bp9DivY4nskF1SIQ45IqfklyTOuHkgTyRF/JqPVrP1pv1/jM6Y+U72+QPrI9vL46bdA=</latexit><latexit sha1_base64="ZfrSnC2Y4QdkDnvbaewdRclM0U=">ACD3icbVC7SgNBFJ2NrxhfUubwajEJuzamEYIKGhnRKNCdgmzk7vJ4OyDmbtCWPYPbPwVGwtFbG3t/BsnMYUmnupwzr3c4+fSKHRtr+swszs3PxCcbG0tLyulZe37jWcao4tHgsY3XrMw1SRNBCgRJuEwUs9CXc+HfHQ/mHpQWcXSFgwS8kPUiEQjO0Eid8t7l6QkFpWJFd47OXQkBVt1AMZ45eXaVu0r0+ri/0ylX7Jo9Ap0mzphUyBjNTvnT7cY8DSFCLpnWbcdO0MuYQsEl5CU31ZAwfsd60DY0YiFoLxv9k9Ndo3RpYDIFcYR0pP7eyFio9SD0zWTIsK8nvaH4n9dOMah7mYiSFCHiP4eCVFKM6bAc2hUKOMqBIYwrYbJS3memDTQVlkwJzuTL0+T6oObYNefioNKoj+soki2yTarEIYekQc5Ik7QIJw/kibyQV+vRerberPef0YI13tkf2B9fAMwLpt2</latexit><latexit sha1_base64="ZfrSnC2Y4QdkDnvbaewdRclM0U=">ACD3icbVC7SgNBFJ2NrxhfUubwajEJuzamEYIKGhnRKNCdgmzk7vJ4OyDmbtCWPYPbPwVGwtFbG3t/BsnMYUmnupwzr3c4+fSKHRtr+swszs3PxCcbG0tLyulZe37jWcao4tHgsY3XrMw1SRNBCgRJuEwUs9CXc+HfHQ/mHpQWcXSFgwS8kPUiEQjO0Eid8t7l6QkFpWJFd47OXQkBVt1AMZ45eXaVu0r0+ri/0ylX7Jo9Ap0mzphUyBjNTvnT7cY8DSFCLpnWbcdO0MuYQsEl5CU31ZAwfsd60DY0YiFoLxv9k9Ndo3RpYDIFcYR0pP7eyFio9SD0zWTIsK8nvaH4n9dOMah7mYiSFCHiP4eCVFKM6bAc2hUKOMqBIYwrYbJS3memDTQVlkwJzuTL0+T6oObYNefioNKoj+soki2yTarEIYekQc5Ik7QIJw/kibyQV+vRerberPef0YI13tkf2B9fAMwLpt2</latexit><latexit sha1_base64="ZfrSnC2Y4QdkDnvbaewdRclM0U=">ACD3icbVC7SgNBFJ2NrxhfUubwajEJuzamEYIKGhnRKNCdgmzk7vJ4OyDmbtCWPYPbPwVGwtFbG3t/BsnMYUmnupwzr3c4+fSKHRtr+swszs3PxCcbG0tLyulZe37jWcao4tHgsY3XrMw1SRNBCgRJuEwUs9CXc+HfHQ/mHpQWcXSFgwS8kPUiEQjO0Eid8t7l6QkFpWJFd47OXQkBVt1AMZ45eXaVu0r0+ri/0ylX7Jo9Ap0mzphUyBjNTvnT7cY8DSFCLpnWbcdO0MuYQsEl5CU31ZAwfsd60DY0YiFoLxv9k9Ndo3RpYDIFcYR0pP7eyFio9SD0zWTIsK8nvaH4n9dOMah7mYiSFCHiP4eCVFKM6bAc2hUKOMqBIYwrYbJS3memDTQVlkwJzuTL0+T6oObYNefioNKoj+soki2yTarEIYekQc5Ik7QIJw/kibyQV+vRerberPef0YI13tkf2B9fAMwLpt2</latexit><latexit sha1_base64="ZfrSnC2Y4QdkDnvbaewdRclM0U=">ACD3icbVC7SgNBFJ2NrxhfUubwajEJuzamEYIKGhnRKNCdgmzk7vJ4OyDmbtCWPYPbPwVGwtFbG3t/BsnMYUmnupwzr3c4+fSKHRtr+swszs3PxCcbG0tLyulZe37jWcao4tHgsY3XrMw1SRNBCgRJuEwUs9CXc+HfHQ/mHpQWcXSFgwS8kPUiEQjO0Eid8t7l6QkFpWJFd47OXQkBVt1AMZ45eXaVu0r0+ri/0ylX7Jo9Ap0mzphUyBjNTvnT7cY8DSFCLpnWbcdO0MuYQsEl5CU31ZAwfsd60DY0YiFoLxv9k9Ndo3RpYDIFcYR0pP7eyFio9SD0zWTIsK8nvaH4n9dOMah7mYiSFCHiP4eCVFKM6bAc2hUKOMqBIYwrYbJS3memDTQVlkwJzuTL0+T6oObYNefioNKoj+soki2yTarEIYekQc5Ik7QIJw/kibyQV+vRerberPef0YI13tkf2B9fAMwLpt2</latexit><latexit sha1_base64="ZfrSnC2Y4QdkDnvbaewdRclM0U=">ACD3icbVC7SgNBFJ2NrxhfUubwajEJuzamEYIKGhnRKNCdgmzk7vJ4OyDmbtCWPYPbPwVGwtFbG3t/BsnMYUmnupwzr3c4+fSKHRtr+swszs3PxCcbG0tLyulZe37jWcao4tHgsY3XrMw1SRNBCgRJuEwUs9CXc+HfHQ/mHpQWcXSFgwS8kPUiEQjO0Eid8t7l6QkFpWJFd47OXQkBVt1AMZ45eXaVu0r0+ri/0ylX7Jo9Ap0mzphUyBjNTvnT7cY8DSFCLpnWbcdO0MuYQsEl5CU31ZAwfsd60DY0YiFoLxv9k9Ndo3RpYDIFcYR0pP7eyFio9SD0zWTIsK8nvaH4n9dOMah7mYiSFCHiP4eCVFKM6bAc2hUKOMqBIYwrYbJS3memDTQVlkwJzuTL0+T6oObYNefioNKoj+soki2yTarEIYekQc5Ik7QIJw/kibyQV+vRerberPef0YI13tkf2B9fAMwLpt2</latexit><latexit sha1_base64="ZfrSnC2Y4QdkDnvbaewdRclM0U=">ACD3icbVC7SgNBFJ2NrxhfUubwajEJuzamEYIKGhnRKNCdgmzk7vJ4OyDmbtCWPYPbPwVGwtFbG3t/BsnMYUmnupwzr3c4+fSKHRtr+swszs3PxCcbG0tLyulZe37jWcao4tHgsY3XrMw1SRNBCgRJuEwUs9CXc+HfHQ/mHpQWcXSFgwS8kPUiEQjO0Eid8t7l6QkFpWJFd47OXQkBVt1AMZ45eXaVu0r0+ri/0ylX7Jo9Ap0mzphUyBjNTvnT7cY8DSFCLpnWbcdO0MuYQsEl5CU31ZAwfsd60DY0YiFoLxv9k9Ndo3RpYDIFcYR0pP7eyFio9SD0zWTIsK8nvaH4n9dOMah7mYiSFCHiP4eCVFKM6bAc2hUKOMqBIYwrYbJS3memDTQVlkwJzuTL0+T6oObYNefioNKoj+soki2yTarEIYekQc5Ik7QIJw/kibyQV+vRerberPef0YI13tkf2B9fAMwLpt2</latexit>
slide-6
SLIDE 6

Our results

  • SGDo error bounds:

[3]: Jeffery Z HaoChen and Suvrit Sra. Random shuffling beats sgd after finite epochs. 2018 [4]: Prateek Jain, Dheeraj Nagaraj, and Praneeth Netrapalli. Sgd without replacement: Sharper rates for general smooth convex functions. 2019 [5]: Itay Safran and Ohad Shamir. How good is sgd with random shuffling?

n = # functions K = # epochs T = nK SGD error = O (1/T )

Upper bound [3,4] 𝑃

! "! + #" "" , 𝑃 # "!

Lower bound [5] Ω

! "! + #! ""

𝑃

! "! + #! ""

F is strongly convex quadratic

O ⇣

1 T 2 + n3 T 3

⌘ , O n

T 2

  • <latexit sha1_base64="qfkrqMBeq96h7LAFThzftfMNWXo=">ACNHicbVDLSgMxFL3j2/qunQTLIvyowVdFlwI7pQwarQqSWTZtrQTGZI7ghlmI9y4e4EcGFIm79BtPHwteBwMk5LcEyRSGHTdZ2dsfGJyanpmtjA3v7C4VFxeuTJxqhmvsVjG+iaghkuheA0FSn6TaE6jQPLroHvU96/vuDYiVpfYS3gjom0lQsEoWqlZPD3zJQ9x0w81ZmXZ5e3eznZGV7VbaUvVHLia9Hu4NYu+RFXo/jIbRZLbtkdgPwl3oiUqtuVk1cAOG8WH/1WzNKIK2SGlP3AQbGdUomOR5wU8NTyjr0javW6poxE0jGydkw2rtEgYa3sUkoH6fSKjkTG9KLDJiGLH/Pb64n9ePcXwsJEJlaTIFRs+FKaSYEz6DZKW0Jyh7FlCmRb2r4R1qO0Dbc8FW4L3e+W/5Gqv7Ll78IrVfdhiBlYg3XYBA8OoArHcA41YHAPT/AKb86D8+K8Ox/D6JgzmlmFH3A+vwDT1qxr</latexit><latexit sha1_base64="Ex4zCwDXobCY9rO0jnWbe/WvfA=">ACNHicbVDLSgMxFM34tr6qLt0Ei1AflBkr6LgRnRhVaFTlsyaYNzWSG5I5QhvkEP8aNH+JGBeKuBO/wfSxsNUDgZNziW5x4sE12DbL9bU9Mzs3PzCYmZpeWV1Lbu+ca3DWFWpaEI1a1HNBNcsipwEOw2UowEnmA3Xve079/cMaV5KCvQi1g9IG3JfU4JGKmZvbh0BfMh7/qK0MRJk0rjMX7w6tsFPtCMcWu4u0O7B7gsbgcxUduM5uzC/YA+C9xRiRX2iuef927UG5mn9xWSOASaCaF1z7AjqCVHAqWBpxo01iwjtkjarGSpJwHQ9GSyd4h2jtLAfKnMk4IH6eyIhgda9wDPJgEBHT3p98T+vFoN/Uk+4jGJgkg4f8mOBIcT9BnGLK0ZB9AwhVHzV0w7xPQBpueMKcGZXPkvuT4sOHbBuXJypSM0xALaQtsojx0jEroDJVRFVH0gJ7RG3q3Hq1X68P6HEanrNHMJhqD9f0Dtmut1A=</latexit><latexit sha1_base64="Ex4zCwDXobCY9rO0jnWbe/WvfA=">ACNHicbVDLSgMxFM34tr6qLt0Ei1AflBkr6LgRnRhVaFTlsyaYNzWSG5I5QhvkEP8aNH+JGBeKuBO/wfSxsNUDgZNziW5x4sE12DbL9bU9Mzs3PzCYmZpeWV1Lbu+ca3DWFWpaEI1a1HNBNcsipwEOw2UowEnmA3Xve079/cMaV5KCvQi1g9IG3JfU4JGKmZvbh0BfMh7/qK0MRJk0rjMX7w6tsFPtCMcWu4u0O7B7gsbgcxUduM5uzC/YA+C9xRiRX2iuef927UG5mn9xWSOASaCaF1z7AjqCVHAqWBpxo01iwjtkjarGSpJwHQ9GSyd4h2jtLAfKnMk4IH6eyIhgda9wDPJgEBHT3p98T+vFoN/Uk+4jGJgkg4f8mOBIcT9BnGLK0ZB9AwhVHzV0w7xPQBpueMKcGZXPkvuT4sOHbBuXJypSM0xALaQtsojx0jEroDJVRFVH0gJ7RG3q3Hq1X68P6HEanrNHMJhqD9f0Dtmut1A=</latexit><latexit sha1_base64="l8ny/Sol9A8RMyoC0QR6pJWBzhc=">ACNHicbVDLSgMxFM34rPU16tJNsAiKUmZaQZcFN4ILK7QqdGrJpJk2NJMZkjtCGeaj3PghbkRwoYhbv8FMOwtfBwIn5xLco8fC67BcZ6tmdm5+YXF0lJ5eWV1bd3e2LzSUaIoa9NIROrGJ5oJLlkbOAh2EytGQl+wa390mvXd0xpHskWjGPWDclA8oBTAkbq2ecXnmAB7HmBIjR1s7R1W8vwfQqb+u5UM+wp/hgCPuH+EdcFvHC7dkVp+pMgP8StyAVKDZsx+9fkSTkEmgmjdcZ0YuilRwKlgWdlLNIsJHZEB6xgqSch0N50sneFdo/RxEClzJOCJ+n0iJaHW49A3yZDAUP/2cvE/r5NAcNJNuYwTYJOHwoSgSHCeYO4zxWjIMaGEKq4+SumQ2L6ANz2ZTg/l75L7mqV2n6l6lcZRUcJbaMdtIdcdIwa6Aw1URtRdI+e0Ct6sx6sF+vd+phGZ6xiZgv9gPX5BW4iqQ=</latexit>

Ω ⇣

1 T 2 + n2 T 3

<latexit sha1_base64="zcUHuqxWNrdGIuDEcEfnHeimzw=">ACHicbZBNSyNBEIZrdP3Y+BX1uJdmg+AHhBkj6DHgZdnLZiFRIRNDT6cmadLTM3TXCGYH+LFv+LFwy6yFw/C3vanbCfxsKu+0PD0W1V01xtlSlry/d/ewuKHpeWV1Y+VtfWNza3q9s6FTXMjsCNSlZqriFtUmOHJCm8ygzyJFJ4GY3Pp/XLGzRWprpNkwx7CR9qGUvByVn9aiP8luCQs1BhTPthbLgogrJoXx+X7Gh+1Y6d0ShZaORwRAf9as2v+zOxtxC8QK152Pj6BwBa/epTOEhFnqAmobi13cDPqFdwQ1IoLCthbjHjYsyH2HWoeYK2V8yWK9mecwYsTo07mtjM/Xei4Im1kyRynQmnkX1dm5rv1bo5xWe9QuosJ9Ri/lCcK0YpmybFBtKgIDVxwIWR7q9MjLhLhFyeFRdC8Hrlt3BxXA/8evA9qDVPYK5V+ASfYR8COIUmfIEWdEDALdzD/jp3XkP3qP3a964L3M7MJ/8p7/AgMUosk=</latexit><latexit sha1_base64="R4f9TkL/7N04gHl3y/sQSwkxVg=">ACHicbZDLSgMxFIYz3q23qks3wSJ4gTJjBV0W3IgbFdoqdGrJpGfaYCYzJGeEMswj+ABufBU3LhRx40LwHXwI09aFtx8CX/5zDsn5g0QKg67oyNT0xOTc/MFubmFxaXisrDROnmkOdxzLWFwEzIWCOgqUcJFoYFEg4Ty4OhzUz69BGxGrGvYTaEWsq0QoOENrtYsV/ySCLqO+hBA3/VAznl5VrvczenO6KosW6OSU1+Lbg+32sWSW3aHon/B+4JSdbty/Hj42m7+Op3Yp5GoJBLZkzTcxNsZUyj4BLygp8aSBi/Yl1oWlQsAtPKhsvldM6HRrG2h6FdOh+n8hYZEw/CmxnxLBnftcG5n+1ZorhQSsTKkRFB89FKaSYkwHSdGO0MBR9i0wroX9K+U9ZhNBm2fBhuD9XvkvNHbLnlv2zrxSdY+MNEPWyDrZJB7ZJ1VyRE5JnXByS+7JI3ly7pwH59l5GbWOV8zq+SHnLdP5ZqkMg=</latexit><latexit sha1_base64="R4f9TkL/7N04gHl3y/sQSwkxVg=">ACHicbZDLSgMxFIYz3q23qks3wSJ4gTJjBV0W3IgbFdoqdGrJpGfaYCYzJGeEMswj+ABufBU3LhRx40LwHXwI09aFtx8CX/5zDsn5g0QKg67oyNT0xOTc/MFubmFxaXisrDROnmkOdxzLWFwEzIWCOgqUcJFoYFEg4Ty4OhzUz69BGxGrGvYTaEWsq0QoOENrtYsV/ySCLqO+hBA3/VAznl5VrvczenO6KosW6OSU1+Lbg+32sWSW3aHon/B+4JSdbty/Hj42m7+Op3Yp5GoJBLZkzTcxNsZUyj4BLygp8aSBi/Yl1oWlQsAtPKhsvldM6HRrG2h6FdOh+n8hYZEw/CmxnxLBnftcG5n+1ZorhQSsTKkRFB89FKaSYkwHSdGO0MBR9i0wroX9K+U9ZhNBm2fBhuD9XvkvNHbLnlv2zrxSdY+MNEPWyDrZJB7ZJ1VyRE5JnXByS+7JI3ly7pwH59l5GbWOV8zq+SHnLdP5ZqkMg=</latexit><latexit sha1_base64="3lfceGoDHZ5YpFUm0IR5NIQvhfg=">ACHicbZDLSgMxFIYzXmu9V26CRZBEcpMK+iy4MadFawWOrVk0jNtMJMZkjNCGeZB3Pgqblwo4saF4NuYXhZq/SHw5T/nkJw/SKQw6Lpfztz8wuLScmGluLq2vrFZ2tq+NnGqOTR5LGPdCpgBKRQ0UaCEVqKBRYGEm+DubFS/uQdtRKyucJhAJ2J9JULBGVqrW6r5FxH0GfUlhHjgh5rxzMuzq9tqTo8mV2XZGrWc+lr0B3jYLZXdijsWnQVvCmUyVaNb+vB7MU8jUMglM6btuQl2MqZRcAl50U8NJIzfsT60LSoWgelk4+Vyum+dHg1jbY9COnZ/TmQsMmYBbYzYjgwf2sj879aO8XwtJMJlaQIik8eClNJMajpGhPaOAohxY18L+lfIBs4mgzbNoQ/D+rjwL19WK51a8S69cP57GUSC7ZI8cEI+ckDo5Jw3SJw8kCfyQl6dR+fZeXPeJ61znRmh/yS8/kNnVGhAg=</latexit>

O ⇣

1 T 2 + n2 T 3

<latexit sha1_base64="zsnR2YrkUAjxtNUkgW37aICuM0=">ACF3icdVDPSxtBFH6r/aFprakevQwNBVth2YkxMbdAL+JFBaNCNg2zk9lkyOzsMvNWCMv+F178V7x4aCle9ebNP8VJ0kJb2g8efPN97zHvfVGmpMUgePSWl+8fPV6ZbXy5u3au/Xq+40zm+aGiy5PVWouImaFklp0UaISF5kRLImUOI8mX2b+aUwVqb6FKeZ6CdspGUsOUMnDar+EQmViHE7jA3jBS2L06/1kuwsntpxJ+yWJDRyNMZPg2ot8OvNZrtNSeA3Gnv7tOUI3Ws1G21C/WCOWufz7uETABwPqg/hMOV5IjRyxazt0SDfsEMSq5EWQlzKzLGJ2wkeo5qlgjbL+Z3leSjU4YkTo0rjWSu/j5RsMTaRK5zoTh2P7tzcR/eb0c4/1+IXWo9B8VGcK4IpmYVEhtIjmrqCONGul0JHzOXCLoKy6EX5eS/5Ozuk9dMie01mnAiuwBR9gGyi0oAMHcAxd4HAFN/ANvnvX3q3w7tbtC5P2c24Q949895V6D/</latexit><latexit sha1_base64="M6iV7NVlCxXJlg0GUEmJ/SVRAow=">ACF3icdVDLSiNBFK32bXxldOmMAg+oOmKMTG7wGxkNioYFdIxVFeqk8Lq6qbq9kBo+hPcufFXZjOLEXE7vwHP8JKoqCiBy6cOude6t4TJFIY8LwnZ2Jyanpmdm6+sLC4tLxS/LF6ZuJUM95ksYz1RUANl0LxJgiQ/CLRnEaB5OfB1c+hf/6bayNidQqDhLcj2lMiFIyClTpF9wj7koew5Yeasozk2elOce746ey3Ap7Ofa16PVhu1MseW65Wq3XCfbcSmX/gNQsIfu1aqWOieuNUGrs7P16vbhuFN89LsxSyOugElqTIt4CbQzqkEwyfOCnxqeUHZFe7xlqaIRN+1sdFeON63SxWGsbSnAI/X9REYjYwZRYDsjCn3z2RuKX3mtFMKDdiZUkgJXbPxRmEoMR6GhLtCcwZyYAlWthdMetTmwjYKAs2hLdL8fkrOwSm8wJKTUqaIw5tI420BYiqIYa6BAdoyZi6Ab9Qf/QnXPr/HXunYdx64TzOrOGPsD5/wJb7KJo</latexit><latexit sha1_base64="M6iV7NVlCxXJlg0GUEmJ/SVRAow=">ACF3icdVDLSiNBFK32bXxldOmMAg+oOmKMTG7wGxkNioYFdIxVFeqk8Lq6qbq9kBo+hPcufFXZjOLEXE7vwHP8JKoqCiBy6cOude6t4TJFIY8LwnZ2Jyanpmdm6+sLC4tLxS/LF6ZuJUM95ksYz1RUANl0LxJgiQ/CLRnEaB5OfB1c+hf/6bayNidQqDhLcj2lMiFIyClTpF9wj7koew5Yeasozk2elOce746ey3Ap7Ofa16PVhu1MseW65Wq3XCfbcSmX/gNQsIfu1aqWOieuNUGrs7P16vbhuFN89LsxSyOugElqTIt4CbQzqkEwyfOCnxqeUHZFe7xlqaIRN+1sdFeON63SxWGsbSnAI/X9REYjYwZRYDsjCn3z2RuKX3mtFMKDdiZUkgJXbPxRmEoMR6GhLtCcwZyYAlWthdMetTmwjYKAs2hLdL8fkrOwSm8wJKTUqaIw5tI420BYiqIYa6BAdoyZi6Ab9Qf/QnXPr/HXunYdx64TzOrOGPsD5/wJb7KJo</latexit><latexit sha1_base64="0FC7/KEQLej4YwY1f0Q0SH/Er+M=">ACF3icdZDLSgMxFIYz3q23qks3wSIowjCptbU7wY07FawKnVoyaYNZjJDckYow7yFG1/FjQtF3OrOtzG9Cr6Q+DPf84hOV+QSGHA8z6cicmp6ZnZufnCwuLS8kpxde3CxKlmvMFiGeurgBouheINECD5VaI5jQLJL4Obo0H98pZrI2J1Dv2EtyLaVSIUjIKN2kX3BPuSh7Dth5qyjOTZ+XU5x7ujq7LeBns59rXo9mCnXSx5brlardcJ9txKZf+A1Kwh+7VqpY6J6w1VQmOdtovfidmacQVMEmNaRIvgVZGNQgmeV7wU8MTym5olzetVTipUN98rxlk06OIy1PQrwMP0+kdHImH4U2M6IQs/8rg3Cv2rNFMKDViZUkgJXbPRQmEoMR5Awh2hOQPZt4YyLexfMetRSwQsyoKF8LUp/t9clF1iyZyR0mFljGMObaBNtI0IqFDdIxOUQMxdIce0BN6du6dR+fFeR21TjmX0Q87bJxOjnzg=</latexit>

Neither upper bound is better than the other!

Our upper bound

slide-7
SLIDE 7

Our results

  • SGDo error bounds:

[4]: Prateek Jain, Dheeraj Nagaraj, and Praneeth Netrapalli. Sgd without replacement: Sharper rates for general smooth convex functions. 2019 [5]: Itay Safran and Ohad Shamir. How good is sgd with random shuffling?

n = # functions K = # epochs T = nK SGD error = O (1/T )

Upper bound [4] 𝑃

# "!

Lower bound [5] Ω

! "! + #! ""

Ω

# "!

F is strongly convex smooth function

O n

T 2

  • <latexit sha1_base64="8cr+os/0oiPKTadAr8zQ7Qibjs=">ACBXicbVA9SwNBEJ2L3/EramKxSDEJtyJoGXAxk6FRAO5GPY2e8mSvb1jd04IRxpB/Cs2ForY+h/s7PwpbhILTXw8Hhvhpl5QSKFQdf9dHJz8wuLS8sr+dW19Y3Nwtb2lYlTzXidxTLWjYAaLoXidRQoeSPRnEaB5NdB/3TkX9ybUSsajhIeCuiXSVCwShaqV0onvuSh1j2Q01ZpoZ7eZwSHwtuj08aBdKbsUdg8wS74eUqsXw/gsALtqFD78TszTiCpmkxjQ9N8FWRjUKJvkw76eGJ5T1aZc3LVU04qaVjb8Ykn2rdEgYa1sKyVj9PZHRyJhBFNjOiGLPTHsj8T+vmWJ40sqESlLkik0WhakGJNRJKQjNGcoB5ZQpoW9lbAetXmgDS5vQ/CmX54lV4cVz614l16pegQTLMu7EZPDiGKpzBdSBwR08wjO8OA/Ok/PqvE1ac87PzA78gfP+DeVAmlQ=</latexit><latexit sha1_base64="5mCJGuLraHgJ0lATAR/k2DkKMfI=">ACBXicbVDJSgNBEO1xjXGLejSHxiDES5gJgh4DXrwZIRtkYujp9CRNenqG7pAGHIRxF/x4sEFT4L/4E38GTvLQRMfFDzeq6KqnhcJrsG2v6yl5ZXVtfXURnpza3tnN7O3X9NhrCir0lCEquERzQSXrAocBGtEipHAE6zu9S/Gfn3AlOahrMAwYq2AdCX3OSVgpHYme+UK5kPe9RWhiRwlZviCLuKd3tw0s7k7I9AV4kzozkSln/7nvw8l5uZz7dTkjgEmgmjdOwIWglRwKlgo7QbaxYR2id1jRUkoDpVjL5YoSPjdLBfqhMScAT9fdEQgKth4FnOgMCPT3vjcX/vGYM/nkr4TKgUk6XeTHAkOIx5HgDleMghgaQqji5lZMe8TkASa4tAnBmX95kdSKBcuONdOrnSKpkihQ3SE8shBZ6iELlEZVRFt+gBPaFn6956tF6t2nrkjWbOUB/YH38AGZdnDQ=</latexit><latexit sha1_base64="5mCJGuLraHgJ0lATAR/k2DkKMfI=">ACBXicbVDJSgNBEO1xjXGLejSHxiDES5gJgh4DXrwZIRtkYujp9CRNenqG7pAGHIRxF/x4sEFT4L/4E38GTvLQRMfFDzeq6KqnhcJrsG2v6yl5ZXVtfXURnpza3tnN7O3X9NhrCir0lCEquERzQSXrAocBGtEipHAE6zu9S/Gfn3AlOahrMAwYq2AdCX3OSVgpHYme+UK5kPe9RWhiRwlZviCLuKd3tw0s7k7I9AV4kzozkSln/7nvw8l5uZz7dTkjgEmgmjdOwIWglRwKlgo7QbaxYR2id1jRUkoDpVjL5YoSPjdLBfqhMScAT9fdEQgKth4FnOgMCPT3vjcX/vGYM/nkr4TKgUk6XeTHAkOIx5HgDleMghgaQqji5lZMe8TkASa4tAnBmX95kdSKBcuONdOrnSKpkihQ3SE8shBZ6iELlEZVRFt+gBPaFn6956tF6t2nrkjWbOUB/YH38AGZdnDQ=</latexit><latexit sha1_base64="wB1jfCoRcymVJ8j2sWQRazLSsWM=">ACBXicbVBNS8NAEN34WetX1KMeFotQLyUpgh4LXrxZoV/QxLZbtqlm03YnQgl9OLFv+LFgyJe/Q/e/Ddu2xy09cHA470ZuYFieAaHOfbWldW9/YLGwVt3d29/btg8OWjlNFWZPGIladgGgmuGRN4CBYJ1GMRIFg7WB0PfXbD0xpHsGjBPmR2QgecgpASP17JNbT7AQyl6oCM3kJGvcVyfYU3whPOeXIqzgx4mbg5KaEc9Z795fVjmkZMAhVE67rJOBnRAGngk2KXqpZQuiIDFjXUEkipv1s9sUEnxmlj8NYmZKAZ+rviYxEWo+jwHRGBIZ60ZuK/3ndFMIrP+MySYFJOl8UpgJDjKeR4D5XjIYG0Ko4uZWTIfE5AEmuKIJwV18eZm0qhXqbh3bql2kcdRQMfoFJWRiy5RDd2gOmoih7RM3pFb9aT9WK9Wx/z1hUrnzlCf2B9/gD+Bpgt</latexit>

Ω ⇣

1 T 2 + n2 T 3

<latexit sha1_base64="zcUHuqxWNrdGIuDEcEfnHeimzw=">ACHicbZBNSyNBEIZrdP3Y+BX1uJdmg+AHhBkj6DHgZdnLZiFRIRNDT6cmadLTM3TXCGYH+LFv+LFwy6yFw/C3vanbCfxsKu+0PD0W1V01xtlSlry/d/ewuKHpeWV1Y+VtfWNza3q9s6FTXMjsCNSlZqriFtUmOHJCm8ygzyJFJ4GY3Pp/XLGzRWprpNkwx7CR9qGUvByVn9aiP8luCQs1BhTPthbLgogrJoXx+X7Gh+1Y6d0ShZaORwRAf9as2v+zOxtxC8QK152Pj6BwBa/epTOEhFnqAmobi13cDPqFdwQ1IoLCthbjHjYsyH2HWoeYK2V8yWK9mecwYsTo07mtjM/Xei4Im1kyRynQmnkX1dm5rv1bo5xWe9QuosJ9Ri/lCcK0YpmybFBtKgIDVxwIWR7q9MjLhLhFyeFRdC8Hrlt3BxXA/8evA9qDVPYK5V+ASfYR8COIUmfIEWdEDALdzD/jp3XkP3qP3a964L3M7MJ/8p7/AgMUosk=</latexit><latexit sha1_base64="R4f9TkL/7N04gHl3y/sQSwkxVg=">ACHicbZDLSgMxFIYz3q23qks3wSJ4gTJjBV0W3IgbFdoqdGrJpGfaYCYzJGeEMswj+ABufBU3LhRx40LwHXwI09aFtx8CX/5zDsn5g0QKg67oyNT0xOTc/MFubmFxaXisrDROnmkOdxzLWFwEzIWCOgqUcJFoYFEg4Ty4OhzUz69BGxGrGvYTaEWsq0QoOENrtYsV/ySCLqO+hBA3/VAznl5VrvczenO6KosW6OSU1+Lbg+32sWSW3aHon/B+4JSdbty/Hj42m7+Op3Yp5GoJBLZkzTcxNsZUyj4BLygp8aSBi/Yl1oWlQsAtPKhsvldM6HRrG2h6FdOh+n8hYZEw/CmxnxLBnftcG5n+1ZorhQSsTKkRFB89FKaSYkwHSdGO0MBR9i0wroX9K+U9ZhNBm2fBhuD9XvkvNHbLnlv2zrxSdY+MNEPWyDrZJB7ZJ1VyRE5JnXByS+7JI3ly7pwH59l5GbWOV8zq+SHnLdP5ZqkMg=</latexit><latexit sha1_base64="R4f9TkL/7N04gHl3y/sQSwkxVg=">ACHicbZDLSgMxFIYz3q23qks3wSJ4gTJjBV0W3IgbFdoqdGrJpGfaYCYzJGeEMswj+ABufBU3LhRx40LwHXwI09aFtx8CX/5zDsn5g0QKg67oyNT0xOTc/MFubmFxaXisrDROnmkOdxzLWFwEzIWCOgqUcJFoYFEg4Ty4OhzUz69BGxGrGvYTaEWsq0QoOENrtYsV/ySCLqO+hBA3/VAznl5VrvczenO6KosW6OSU1+Lbg+32sWSW3aHon/B+4JSdbty/Hj42m7+Op3Yp5GoJBLZkzTcxNsZUyj4BLygp8aSBi/Yl1oWlQsAtPKhsvldM6HRrG2h6FdOh+n8hYZEw/CmxnxLBnftcG5n+1ZorhQSsTKkRFB89FKaSYkwHSdGO0MBR9i0wroX9K+U9ZhNBm2fBhuD9XvkvNHbLnlv2zrxSdY+MNEPWyDrZJB7ZJ1VyRE5JnXByS+7JI3ly7pwH59l5GbWOV8zq+SHnLdP5ZqkMg=</latexit><latexit sha1_base64="3lfceGoDHZ5YpFUm0IR5NIQvhfg=">ACHicbZDLSgMxFIYzXmu9V26CRZBEcpMK+iy4MadFawWOrVk0jNtMJMZkjNCGeZB3Pgqblwo4saF4NuYXhZq/SHw5T/nkJw/SKQw6Lpfztz8wuLScmGluLq2vrFZ2tq+NnGqOTR5LGPdCpgBKRQ0UaCEVqKBRYGEm+DubFS/uQdtRKyucJhAJ2J9JULBGVqrW6r5FxH0GfUlhHjgh5rxzMuzq9tqTo8mV2XZGrWc+lr0B3jYLZXdijsWnQVvCmUyVaNb+vB7MU8jUMglM6btuQl2MqZRcAl50U8NJIzfsT60LSoWgelk4+Vyum+dHg1jbY9COnZ/TmQsMmYBbYzYjgwf2sj879aO8XwtJMJlaQIik8eClNJMajpGhPaOAohxY18L+lfIBs4mgzbNoQ/D+rjwL19WK51a8S69cP57GUSC7ZI8cEI+ckDo5Jw3SJw8kCfyQl6dR+fZeXPeJ61znRmh/yS8/kNnVGhAg=</latexit>

Ω n

T 2

  • <latexit sha1_base64="3H9J7NjN6NpHuIYu0Npd0bPw8cI=">AC3icdVBSxtBFH6rtqaprbE96mEwCPYSdkNMzC3QS2+NYFTIpmF28jYZnJ1dZt4KYclR6MW/0osHRXr1D/TmzZ/iJGmhLfrBg4/ve4/3hdlSlry/QdvZXt1ev10pvy24137zcrWx9ObJobgT2RqtScRdyikhp7JEnhWaQJ5HC0+j89w/vUBjZaqPaZrhIOFjLWMpODlpWNkNvyY45ixUGN+GBsuCj0rjr/VZyw0cjyhT8NK1a/Vm812O2B+rdE4OAxajgQHrWajzYKav0C1sxNfPgJAd1j5FY5SkSeoShubT/wMxoU3JAUCmflMLeYcXHOx9h3VPME7aBY/DJje04ZsTg1rjSxhfr3RMETa6dJ5DoThP7vzcXn/P6OcWHg0LqLCfUYrkozhWjlM2DYSNpUJCaOsKFke5WJibc5UEuvrIL4c+n7GVyUq8FLpmjoNpwBIl2IZd2IcAWtCBL9CFHgj4Dj/gBm69K+/au/N+LltXvN8zH+EfePdPtmidBA=</latexit><latexit sha1_base64="EPfFL7uvC5XNZOj9kltJ2qh+Iik=">AC3icdVBNSyNBEO1RV2PW1ahHPTQGwb2EmRDzcQt48aCUSGTDT2dmqRJT8/QXSOEIUfBi3/FiwdFvPoHvIl/xk6isLvog4LHe1VU1QsSKQy67qszN7/wY3Ept5z/ufJrda2wvnFm4lRzaPFYxvoiYAakUNBCgRIuEg0sCiScB8ODiX9+CdqIWJ3iKIFOxPpKhIztFK3sOMfRdBn1JcQ4p4fasYzNc5O/5TH1NeiP8Df3ULRLZWr1UbDo26pUtmvezVLvP1atdKgXsmdotjcDq/eLu+ejruF78X8zQChVwyY9qem2AnYxoFlzDO+6mBhPEh60PbUsUiMJ1s+suY7lqlR8NY21JIp+rfExmLjBlFge2MGA7M/95E/MprpxjWO5lQSYqg+GxRmEqKMZ0EQ3tCA0c5soRxLeytlA+YzQNtfHkbwuen9HtyVi5NpkTr9iskBlyZIvskD3ikRpkNyTFqEk2tyS+7Jg3Pj3DmPztOsdc75mNk/8B5fgc3hZ7k</latexit><latexit sha1_base64="EPfFL7uvC5XNZOj9kltJ2qh+Iik=">AC3icdVBNSyNBEO1RV2PW1ahHPTQGwb2EmRDzcQt48aCUSGTDT2dmqRJT8/QXSOEIUfBi3/FiwdFvPoHvIl/xk6isLvog4LHe1VU1QsSKQy67qszN7/wY3Ept5z/ufJrda2wvnFm4lRzaPFYxvoiYAakUNBCgRIuEg0sCiScB8ODiX9+CdqIWJ3iKIFOxPpKhIztFK3sOMfRdBn1JcQ4p4fasYzNc5O/5TH1NeiP8Df3ULRLZWr1UbDo26pUtmvezVLvP1atdKgXsmdotjcDq/eLu+ejruF78X8zQChVwyY9qem2AnYxoFlzDO+6mBhPEh60PbUsUiMJ1s+suY7lqlR8NY21JIp+rfExmLjBlFge2MGA7M/95E/MprpxjWO5lQSYqg+GxRmEqKMZ0EQ3tCA0c5soRxLeytlA+YzQNtfHkbwuen9HtyVi5NpkTr9iskBlyZIvskD3ikRpkNyTFqEk2tyS+7Jg3Pj3DmPztOsdc75mNk/8B5fgc3hZ7k</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="AlZ7nNKAw53EaOL3UNpuw+eTjeU=">ACAHicbZC9SgNBFIXvxv/4F21thgQhNmE3jZaCjZ0KRgPZGYnd5PB2dl5q4QlvQ2voqNhSI+hZ1v4ySm0MQDA4dzZrhzvyhT0pLvf3mlpeWV1bX1jfLm1vbObmVv68amuRHYEqlKTviFpXU2CJCtuZQZ5ECm+j+7NJf/uAxspUX9Mow27CB1rGUnByUa9SDS8SHAWKoypHsaGi0KPi+u75piFRg6GdNSr1PyGPxVbNMHM1GCmy17lM+ynIk9Qk1Dc2k7gZ9QtuCEpFI7LYW4x4+KeD7DjrOYJ2m4x3WXMDl3SZ3Fq3NHEpunvFwVPrB0lkbuZcBra+W4S/td1copPuoXUWU6oxc+gOFeMUjYBw/rSoCA1coYLI91fmRhyx4McvrKDEMyvGhumo3AbwRXPqzDAVShDgEcwymcwyW0QMAjPMrvHlP3ov3/oOr5M247cMfeR/fqJOZGg=</latexit><latexit sha1_base64="buvRZctVPUoqahVPmGp0U5RwU2U=">ACAHicdZBPaxpBGMbftf9Saxrbay6DUrAX2RWj8VboJbcaiEZwrcyO7+rg7Owy825BFu+5Kv0kNCyafILd8mo7GQlPSBgYfnmeGd9xdlSlry/Xuv9Or1m7fv9t6XP1T2Px5UP1WGNs2NwIFIVWpGEbeopMYBSVI4ygzyJFJ4Hi2/b/rzX2isTPUZrTKcJHyuZSwFJxdNq7XwR4JzkKFMTXC2HBR6HVx9rO1ZqGR8wV9nVbrfrPV6fR6AfOb7fbRcdB1Jjqdto9FjT9reqwU39avQtnqcgT1CQUt3Yc+BlNCm5ICoXrcphbzLhY8jmOndU8QTsptrus2ReXzFicGnc0sW369EXBE2tXSeRuJpwW9t9uE7UjXOKjyeF1FlOqMXjoDhXjFK2AcNm0qAgtXKGCyPdX5lYcMeDHL6yg/B3U/Z/M2w1A0fm1Ic9OIQaNCALnyDE+jDARcwG+4hv0rvy/jziKnk7bp/hmbzbBySmXg=</latexit><latexit sha1_base64="aYIc/keXt0K0kT+akKw7EL9Ad3I=">AC3icdZA9SwNBEIb3/IzxK2psxgEbcJdyIfpBs7IxgVcjHsbeaSxb29Y3dOCEd6G/+KjYUitv4BO/+NmxhBRV9YeHlnhtl5gkQKg67szMzs0vLOaW8srq2vrhY3NcxOnmkOLxzLWlwEzIWCFgqUcJloYFEg4SK4PhrXL25AGxGrMxwm0IlYX4lQcIY26hZ2/JMI+oz6EkLc80PNeKZG2dlVeUR9LfoD3O8Wim6pXKs1Gh51S5VK9cCrW+NV67VKg3old6IimarZLbz5vZinESjkhnT9twEOxnTKLiEUd5PDSMX7M+tK1VLALTySa3jOiuTXo0jLV9Cuk/T6RsciYRTYzojhwPyujcO/au0Uw4NOJlSIij+uShMJcWYjsHQntDAUQ6tYVwL+1fKB8zyQIsvbyF8XUr/N+flkmfJnLrFw8oUR45skx2yRzxSJ4fkmDRJi3ByS+7JI3ly7pwH59l5+WydcaYzW+SHnNcPzt6a3A=</latexit><latexit sha1_base64="eM7aLkvAQ9Tvc4wx/mDon6W6MuE=">AC3icdZA9SwNBEIb3/IzxK2psxgEbcJdyIfpBs7IxgVcjHsbeaSxb29Y3dOCEd6G/+KjYUitv4BO/+NmxhBRV9YeHlnhtl5gkQKg67szMzs0vLOaW8srq2vrhY3NcxOnmkOLxzLWlwEzIWCFgqUcJloYFEg4SK4PhrXL25AGxGrMxwm0IlYX4lQcIY26hZ2/JMI+oz6EkLc80PNeKZG2dlVeUR9LfoD3O8Wim6pXKs1Gh51S5VK9cCrW+NV67VKg3old6IimarZLbz5vZinESjkhnT9twEOxnTKLiEUd5PDSMX7M+tK1VLALTySa3jOiuTXo0jLV9Cuk/T6RsciYRTYzojhwPyujcO/au0Uw4NOJlSIij+uShMJcWYjsHQntDAUQ6tYVwL+1fKB8zyQIsvbyF8XUr/N+flkmfJnHrFw8oUR45skx2yRzxSJ4fkmDRJi3ByS+7JI3ly7pwH59l5+WydcaYzW+SHnNcPzy6a3Q=</latexit><latexit sha1_base64="EPfFL7uvC5XNZOj9kltJ2qh+Iik=">AC3icdVBNSyNBEO1RV2PW1ahHPTQGwb2EmRDzcQt48aCUSGTDT2dmqRJT8/QXSOEIUfBi3/FiwdFvPoHvIl/xk6isLvog4LHe1VU1QsSKQy67qszN7/wY3Ept5z/ufJrda2wvnFm4lRzaPFYxvoiYAakUNBCgRIuEg0sCiScB8ODiX9+CdqIWJ3iKIFOxPpKhIztFK3sOMfRdBn1JcQ4p4fasYzNc5O/5TH1NeiP8Df3ULRLZWr1UbDo26pUtmvezVLvP1atdKgXsmdotjcDq/eLu+ejruF78X8zQChVwyY9qem2AnYxoFlzDO+6mBhPEh60PbUsUiMJ1s+suY7lqlR8NY21JIp+rfExmLjBlFge2MGA7M/95E/MprpxjWO5lQSYqg+GxRmEqKMZ0EQ3tCA0c5soRxLeytlA+YzQNtfHkbwuen9HtyVi5NpkTr9iskBlyZIvskD3ikRpkNyTFqEk2tyS+7Jg3Pj3DmPztOsdc75mNk/8B5fgc3hZ7k</latexit><latexit sha1_base64="EPfFL7uvC5XNZOj9kltJ2qh+Iik=">AC3icdVBNSyNBEO1RV2PW1ahHPTQGwb2EmRDzcQt48aCUSGTDT2dmqRJT8/QXSOEIUfBi3/FiwdFvPoHvIl/xk6isLvog4LHe1VU1QsSKQy67qszN7/wY3Ept5z/ufJrda2wvnFm4lRzaPFYxvoiYAakUNBCgRIuEg0sCiScB8ODiX9+CdqIWJ3iKIFOxPpKhIztFK3sOMfRdBn1JcQ4p4fasYzNc5O/5TH1NeiP8Df3ULRLZWr1UbDo26pUtmvezVLvP1atdKgXsmdotjcDq/eLu+ejruF78X8zQChVwyY9qem2AnYxoFlzDO+6mBhPEh60PbUsUiMJ1s+suY7lqlR8NY21JIp+rfExmLjBlFge2MGA7M/95E/MprpxjWO5lQSYqg+GxRmEqKMZ0EQ3tCA0c5soRxLeytlA+YzQNtfHkbwuen9HtyVi5NpkTr9iskBlyZIvskD3ikRpkNyTFqEk2tyS+7Jg3Pj3DmPztOsdc75mNk/8B5fgc3hZ7k</latexit><latexit sha1_base64="EPfFL7uvC5XNZOj9kltJ2qh+Iik=">AC3icdVBNSyNBEO1RV2PW1ahHPTQGwb2EmRDzcQt48aCUSGTDT2dmqRJT8/QXSOEIUfBi3/FiwdFvPoHvIl/xk6isLvog4LHe1VU1QsSKQy67qszN7/wY3Ept5z/ufJrda2wvnFm4lRzaPFYxvoiYAakUNBCgRIuEg0sCiScB8ODiX9+CdqIWJ3iKIFOxPpKhIztFK3sOMfRdBn1JcQ4p4fasYzNc5O/5TH1NeiP8Df3ULRLZWr1UbDo26pUtmvezVLvP1atdKgXsmdotjcDq/eLu+ejruF78X8zQChVwyY9qem2AnYxoFlzDO+6mBhPEh60PbUsUiMJ1s+suY7lqlR8NY21JIp+rfExmLjBlFge2MGA7M/95E/MprpxjWO5lQSYqg+GxRmEqKMZ0EQ3tCA0c5soRxLeytlA+YzQNtfHkbwuen9HtyVi5NpkTr9iskBlyZIvskD3ikRpkNyTFqEk2tyS+7Jg3Pj3DmPztOsdc75mNk/8B5fgc3hZ7k</latexit><latexit sha1_base64="EPfFL7uvC5XNZOj9kltJ2qh+Iik=">AC3icdVBNSyNBEO1RV2PW1ahHPTQGwb2EmRDzcQt48aCUSGTDT2dmqRJT8/QXSOEIUfBi3/FiwdFvPoHvIl/xk6isLvog4LHe1VU1QsSKQy67qszN7/wY3Ept5z/ufJrda2wvnFm4lRzaPFYxvoiYAakUNBCgRIuEg0sCiScB8ODiX9+CdqIWJ3iKIFOxPpKhIztFK3sOMfRdBn1JcQ4p4fasYzNc5O/5TH1NeiP8Df3ULRLZWr1UbDo26pUtmvezVLvP1atdKgXsmdotjcDq/eLu+ejruF78X8zQChVwyY9qem2AnYxoFlzDO+6mBhPEh60PbUsUiMJ1s+suY7lqlR8NY21JIp+rfExmLjBlFge2MGA7M/95E/MprpxjWO5lQSYqg+GxRmEqKMZ0EQ3tCA0c5soRxLeytlA+YzQNtfHkbwuen9HtyVi5NpkTr9iskBlyZIvskD3ikRpkNyTFqEk2tyS+7Jg3Pj3DmPztOsdc75mNk/8B5fgc3hZ7k</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="AlZ7nNKAw53EaOL3UNpuw+eTjeU=">ACAHicbZC9SgNBFIXvxv/4F21thgQhNmE3jZaCjZ0KRgPZGYnd5PB2dl5q4QlvQ2voqNhSI+hZ1v4ySm0MQDA4dzZrhzvyhT0pLvf3mlpeWV1bX1jfLm1vbObmVv68amuRHYEqlKTviFpXU2CJCtuZQZ5ECm+j+7NJf/uAxspUX9Mow27CB1rGUnByUa9SDS8SHAWKoypHsaGi0KPi+u75piFRg6GdNSr1PyGPxVbNMHM1GCmy17lM+ynIk9Qk1Dc2k7gZ9QtuCEpFI7LYW4x4+KeD7DjrOYJ2m4x3WXMDl3SZ3Fq3NHEpunvFwVPrB0lkbuZcBra+W4S/td1copPuoXUWU6oxc+gOFeMUjYBw/rSoCA1coYLI91fmRhyx4McvrKDEMyvGhumo3AbwRXPqzDAVShDgEcwymcwyW0QMAjPMrvHlP3ov3/oOr5M247cMfeR/fqJOZGg=</latexit><latexit sha1_base64="buvRZctVPUoqahVPmGp0U5RwU2U=">ACAHicdZBPaxpBGMbftf9Saxrbay6DUrAX2RWj8VboJbcaiEZwrcyO7+rg7Owy825BFu+5Kv0kNCyafILd8mo7GQlPSBgYfnmeGd9xdlSlry/Xuv9Or1m7fv9t6XP1T2Px5UP1WGNs2NwIFIVWpGEbeopMYBSVI4ygzyJFJ4Hi2/b/rzX2isTPUZrTKcJHyuZSwFJxdNq7XwR4JzkKFMTXC2HBR6HVx9rO1ZqGR8wV9nVbrfrPV6fR6AfOb7fbRcdB1Jjqdto9FjT9reqwU39avQtnqcgT1CQUt3Yc+BlNCm5ICoXrcphbzLhY8jmOndU8QTsptrus2ReXzFicGnc0sW369EXBE2tXSeRuJpwW9t9uE7UjXOKjyeF1FlOqMXjoDhXjFK2AcNm0qAgtXKGCyPdX5lYcMeDHL6yg/B3U/Z/M2w1A0fm1Ic9OIQaNCALnyDE+jDARcwG+4hv0rvy/jziKnk7bp/hmbzbBySmXg=</latexit><latexit sha1_base64="aYIc/keXt0K0kT+akKw7EL9Ad3I=">AC3icdZA9SwNBEIb3/IzxK2psxgEbcJdyIfpBs7IxgVcjHsbeaSxb29Y3dOCEd6G/+KjYUitv4BO/+NmxhBRV9YeHlnhtl5gkQKg67szMzs0vLOaW8srq2vrhY3NcxOnmkOLxzLWlwEzIWCFgqUcJloYFEg4SK4PhrXL25AGxGrMxwm0IlYX4lQcIY26hZ2/JMI+oz6EkLc80PNeKZG2dlVeUR9LfoD3O8Wim6pXKs1Gh51S5VK9cCrW+NV67VKg3old6IimarZLbz5vZinESjkhnT9twEOxnTKLiEUd5PDSMX7M+tK1VLALTySa3jOiuTXo0jLV9Cuk/T6RsciYRTYzojhwPyujcO/au0Uw4NOJlSIij+uShMJcWYjsHQntDAUQ6tYVwL+1fKB8zyQIsvbyF8XUr/N+flkmfJnLrFw8oUR45skx2yRzxSJ4fkmDRJi3ByS+7JI3ly7pwH59l5+WydcaYzW+SHnNcPzt6a3A=</latexit><latexit sha1_base64="eM7aLkvAQ9Tvc4wx/mDon6W6MuE=">AC3icdZA9SwNBEIb3/IzxK2psxgEbcJdyIfpBs7IxgVcjHsbeaSxb29Y3dOCEd6G/+KjYUitv4BO/+NmxhBRV9YeHlnhtl5gkQKg67szMzs0vLOaW8srq2vrhY3NcxOnmkOLxzLWlwEzIWCFgqUcJloYFEg4SK4PhrXL25AGxGrMxwm0IlYX4lQcIY26hZ2/JMI+oz6EkLc80PNeKZG2dlVeUR9LfoD3O8Wim6pXKs1Gh51S5VK9cCrW+NV67VKg3old6IimarZLbz5vZinESjkhnT9twEOxnTKLiEUd5PDSMX7M+tK1VLALTySa3jOiuTXo0jLV9Cuk/T6RsciYRTYzojhwPyujcO/au0Uw4NOJlSIij+uShMJcWYjsHQntDAUQ6tYVwL+1fKB8zyQIsvbyF8XUr/N+flkmfJnHrFw8oUR45skx2yRzxSJ4fkmDRJi3ByS+7JI3ly7pwH59l5+WydcaYzW+SHnNcPzy6a3Q=</latexit><latexit sha1_base64="EPfFL7uvC5XNZOj9kltJ2qh+Iik=">AC3icdVBNSyNBEO1RV2PW1ahHPTQGwb2EmRDzcQt48aCUSGTDT2dmqRJT8/QXSOEIUfBi3/FiwdFvPoHvIl/xk6isLvog4LHe1VU1QsSKQy67qszN7/wY3Ept5z/ufJrda2wvnFm4lRzaPFYxvoiYAakUNBCgRIuEg0sCiScB8ODiX9+CdqIWJ3iKIFOxPpKhIztFK3sOMfRdBn1JcQ4p4fasYzNc5O/5TH1NeiP8Df3ULRLZWr1UbDo26pUtmvezVLvP1atdKgXsmdotjcDq/eLu+ejruF78X8zQChVwyY9qem2AnYxoFlzDO+6mBhPEh60PbUsUiMJ1s+suY7lqlR8NY21JIp+rfExmLjBlFge2MGA7M/95E/MprpxjWO5lQSYqg+GxRmEqKMZ0EQ3tCA0c5soRxLeytlA+YzQNtfHkbwuen9HtyVi5NpkTr9iskBlyZIvskD3ikRpkNyTFqEk2tyS+7Jg3Pj3DmPztOsdc75mNk/8B5fgc3hZ7k</latexit><latexit sha1_base64="EPfFL7uvC5XNZOj9kltJ2qh+Iik=">AC3icdVBNSyNBEO1RV2PW1ahHPTQGwb2EmRDzcQt48aCUSGTDT2dmqRJT8/QXSOEIUfBi3/FiwdFvPoHvIl/xk6isLvog4LHe1VU1QsSKQy67qszN7/wY3Ept5z/ufJrda2wvnFm4lRzaPFYxvoiYAakUNBCgRIuEg0sCiScB8ODiX9+CdqIWJ3iKIFOxPpKhIztFK3sOMfRdBn1JcQ4p4fasYzNc5O/5TH1NeiP8Df3ULRLZWr1UbDo26pUtmvezVLvP1atdKgXsmdotjcDq/eLu+ejruF78X8zQChVwyY9qem2AnYxoFlzDO+6mBhPEh60PbUsUiMJ1s+suY7lqlR8NY21JIp+rfExmLjBlFge2MGA7M/95E/MprpxjWO5lQSYqg+GxRmEqKMZ0EQ3tCA0c5soRxLeytlA+YzQNtfHkbwuen9HtyVi5NpkTr9iskBlyZIvskD3ikRpkNyTFqEk2tyS+7Jg3Pj3DmPztOsdc75mNk/8B5fgc3hZ7k</latexit><latexit sha1_base64="EPfFL7uvC5XNZOj9kltJ2qh+Iik=">AC3icdVBNSyNBEO1RV2PW1ahHPTQGwb2EmRDzcQt48aCUSGTDT2dmqRJT8/QXSOEIUfBi3/FiwdFvPoHvIl/xk6isLvog4LHe1VU1QsSKQy67qszN7/wY3Ept5z/ufJrda2wvnFm4lRzaPFYxvoiYAakUNBCgRIuEg0sCiScB8ODiX9+CdqIWJ3iKIFOxPpKhIztFK3sOMfRdBn1JcQ4p4fasYzNc5O/5TH1NeiP8Df3ULRLZWr1UbDo26pUtmvezVLvP1atdKgXsmdotjcDq/eLu+ejruF78X8zQChVwyY9qem2AnYxoFlzDO+6mBhPEh60PbUsUiMJ1s+suY7lqlR8NY21JIp+rfExmLjBlFge2MGA7M/95E/MprpxjWO5lQSYqg+GxRmEqKMZ0EQ3tCA0c5soRxLeytlA+YzQNtfHkbwuen9HtyVi5NpkTr9iskBlyZIvskD3ikRpkNyTFqEk2tyS+7Jg3Pj3DmPztOsdc75mNk/8B5fgc3hZ7k</latexit><latexit sha1_base64="EPfFL7uvC5XNZOj9kltJ2qh+Iik=">AC3icdVBNSyNBEO1RV2PW1ahHPTQGwb2EmRDzcQt48aCUSGTDT2dmqRJT8/QXSOEIUfBi3/FiwdFvPoHvIl/xk6isLvog4LHe1VU1QsSKQy67qszN7/wY3Ept5z/ufJrda2wvnFm4lRzaPFYxvoiYAakUNBCgRIuEg0sCiScB8ODiX9+CdqIWJ3iKIFOxPpKhIztFK3sOMfRdBn1JcQ4p4fasYzNc5O/5TH1NeiP8Df3ULRLZWr1UbDo26pUtmvezVLvP1atdKgXsmdotjcDq/eLu+ejruF78X8zQChVwyY9qem2AnYxoFlzDO+6mBhPEh60PbUsUiMJ1s+suY7lqlR8NY21JIp+rfExmLjBlFge2MGA7M/95E/MprpxjWO5lQSYqg+GxRmEqKMZ0EQ3tCA0c5soRxLeytlA+YzQNtfHkbwuen9HtyVi5NpkTr9iskBlyZIvskD3ikRpkNyTFqEk2tyS+7Jg3Pj3DmPztOsdc75mNk/8B5fgc3hZ7k</latexit><latexit sha1_base64="eM7aLkvAQ9Tvc4wx/mDon6W6MuE=">AC3icdZA9SwNBEIb3/IzxK2psxgEbcJdyIfpBs7IxgVcjHsbeaSxb29Y3dOCEd6G/+KjYUitv4BO/+NmxhBRV9YeHlnhtl5gkQKg67szMzs0vLOaW8srq2vrhY3NcxOnmkOLxzLWlwEzIWCFgqUcJloYFEg4SK4PhrXL25AGxGrMxwm0IlYX4lQcIY26hZ2/JMI+oz6EkLc80PNeKZG2dlVeUR9LfoD3O8Wim6pXKs1Gh51S5VK9cCrW+NV67VKg3old6IimarZLbz5vZinESjkhnT9twEOxnTKLiEUd5PDSMX7M+tK1VLALTySa3jOiuTXo0jLV9Cuk/T6RsciYRTYzojhwPyujcO/au0Uw4NOJlSIij+uShMJcWYjsHQntDAUQ6tYVwL+1fKB8zyQIsvbyF8XUr/N+flkmfJnHrFw8oUR45skx2yRzxSJ4fkmDRJi3ByS+7JI3ly7pwH59l5+WydcaYzW+SHnNcPzy6a3Q=</latexit>

Our lower bound

Surprisingly, lower bound is different for non-quadratics!

slide-8
SLIDE 8

Upper bound

slide-9
SLIDE 9

Upper bound - Approach

  • x1 = Start of epoch, xn = End of epoch
  • Idea [3]:

𝑦! − 𝑦" = 𝛽 %

#

∇𝑔

$(#) 𝑦# ≈ % #

∇𝑔

$(#) 𝑦" = 𝛽𝑜∇𝐺 𝑦"

  • Key lemma [4] : E ||𝑦# − 𝑦"||' = 𝑃(𝑗𝛽')
  • (Tight!)

n steps of gradient descent!

[3]: Jeffery Z HaoChen and Suvrit Sra. Random shuffling beats sgd after finite epochs. 2018 [4]: Prateek Jain, Dheeraj Nagaraj, and Praneeth Netrapalli. Sgd without replacement: Sharper rates for general smooth convex functions. 2019

E[kxi x1k2] = O(iα2)

<latexit sha1_base64="ow6FnlNRMZhyIMT8xmWaCmQTW+E=">ACE3icbVC7SgNBFL3rM8ZX1EawGQxCFAy7aUwjBESwM4JhOy6zE4myZDZBzOzkrjJP9j4KzYWitja2Al+jLNJCl8HBg7n3Mudc7yIM6lM8OYmZ2bX1jMLGWXV1bX1nMbm3UZxoLQGgl5K48LClnAa0pji9igTFvsdpw+udpH7jhgrJwuBSDSLq+LgTsDYjWGnJzR3YPlZdz0tOR0172HfZYd+1kD28LiHn+LzAbMyjLkbXpf2sm8ubRXM9JdYU5KvbN9+gkbVzb3brZDEPg0U4VjKpmVGykmwUIxwOsrasaQRJj3coU1NA+xT6STjTCO0p5UWaodCv0Chsfp9I8G+lAPf05NpAvnbS8X/vGas2mUnYUEUKxqQyaF2zJEKUVoQajFBieIDTARTP8VkS4WmChdY1qC9TvyX1IvFS2zaF1Y+UoZJsjADuxCASw4gqcQRVqQOAOHuAJno1749F4MV4nozPGdGcLfsB4+wJxTJ4G</latexit><latexit sha1_base64="So854qD0si5tkRmeh+5dt0zQpAE=">ACE3icbVDLSsNAFJ3UV1tfUTeCm8EiVMGSdGM3QlEd1awD0jSMJlO7dDJg5mJtKYFP8GNP+LCjQtF3LpxJ/gxJm0X2npg4HDOvdw5xwkYFVLTvpTU3PzC4lI6k1eWV1bVzc2a8IPOSZV7DOfNxwkCKMeqUoqGWkEnCDXYaTudE8Tv35DuKC+dyX7AbFcdO3RNsVIxpKtHpgukh3Hic6Ghjno2fSwZ+vQHDSL0Dq+yFMTsaCDYLO4n7XVnFbQRoCzRJ+QXHn79jtz93hSsdVPs+Xj0CWexAwJYehaIK0IcUkxI8OsGQoSINxF18SIqYdcIqxolGkI92KlBds+j58n4Uj9vREhV4i+68STSQIx7SXif54RynbJiqgXhJ4eHyoHTIofZgUBFuUEyxZPyYIcxr/FeIO4gjLuMakBH068iypFQu6VtAv9Vy5BMZIgx2wC/JAB0egDM5BVQBvfgCbyAV+VBeVbelPfxaEqZ7GyBP1A+fgBuJ+D</latexit><latexit sha1_base64="So854qD0si5tkRmeh+5dt0zQpAE=">ACE3icbVDLSsNAFJ3UV1tfUTeCm8EiVMGSdGM3QlEd1awD0jSMJlO7dDJg5mJtKYFP8GNP+LCjQtF3LpxJ/gxJm0X2npg4HDOvdw5xwkYFVLTvpTU3PzC4lI6k1eWV1bVzc2a8IPOSZV7DOfNxwkCKMeqUoqGWkEnCDXYaTudE8Tv35DuKC+dyX7AbFcdO3RNsVIxpKtHpgukh3Hic6Ghjno2fSwZ+vQHDSL0Dq+yFMTsaCDYLO4n7XVnFbQRoCzRJ+QXHn79jtz93hSsdVPs+Xj0CWexAwJYehaIK0IcUkxI8OsGQoSINxF18SIqYdcIqxolGkI92KlBds+j58n4Uj9vREhV4i+68STSQIx7SXif54RynbJiqgXhJ4eHyoHTIofZgUBFuUEyxZPyYIcxr/FeIO4gjLuMakBH068iypFQu6VtAv9Vy5BMZIgx2wC/JAB0egDM5BVQBvfgCbyAV+VBeVbelPfxaEqZ7GyBP1A+fgBuJ+D</latexit><latexit sha1_base64="kR8j2H6CRibCFwo3/bYSGys3B0k=">ACE3icbVDLSsNAFJ3UV62vqEs3g0WogiXTjd0IBRHcWcE+oEnDZDph04ezEykJe0/uPFX3LhQxK0bd/6NSZuFth4YOJxzL3fOcULOpDKMby23srq2vpHfLGxt7+zu6fsHTRlEgtAGCXg2g6WlDOfNhRTnLZDQbHncNpyhlep3qgQrLAv1fjkFoe7vMZQSrRL1M9PDauA48fW0Y05GNjsf2Qiak24FWpe3JWZiHg4w7FZOC7ZeNMrGDHCZoIwUQYa6rX+ZvYBEHvUV4VjKDjJCZcVYKEY4nRbMSNIQkyHu05CfexRacWzTFN4kig96AYieb6CM/X3Row9Kcek0ymCeSil4r/eZ1IuVUrZn4YKeqT+SE34lAFMC0I9pigRPFxQjARLPkrJAMsMFJjWkJaDHyMmlWysgoztUrFWzOvLgCByDEkDgAtTADaiDBiDgETyDV/CmPWkv2rv2MR/NadnOIfgD7fMHLEacWA=</latexit>

kxi x1k2 grows as iα2 instead of i2α2

<latexit sha1_base64="HJu3EKZUqh70iWyC3UFf2orPcI=">ACM3icbZDPahRBEMZrEjVx/bdG8OKlcRG8uMzsIXpc8BI8RXCTwM5mqOmt2W3S0z1015hdJvsqwUfw4ot4EMSDIl59B3t3I2riBw0f36+K6q80spzH+ONjavXb+xtX2zdev2nbv32vd3DrytnaSBtNq6oxw9aWVowIo1HVWOsMw1HeYnL5f8C05r6x5w/OKRiVOjCqURA5R1n6Vns0y9WyWJSI9O+6JlGnGjZg4e+oFerEQKkVdTfEPU8Yz4VjYkmPe7951u7E3XglcdUkF6bTf1idvwOA/az9MR1bWZdkWGr0fpjEFY8adKykpkUrT1VKE9wQsNgDZbkR81q54V4EpKxKwLz7BYpX93NFh6Py/zUFkiT/1ltgz/x4Y1Fy9GjTJVzWTkelBRa8FWLA8oxsqRZD0PBqVT4a9CTtGh5HDmVjhCcnlq+ag103ibvI6fR3Ya1teASP4Sk8Bz6sAf7MAJ7+ETfIVv0YfoS/Q9+rEu3Ygueh7AP4p+/gKlPKvU</latexit><latexit sha1_base64="vtehmgvKW9+ibfNlf7CzDKgGgWA=">ACM3icbZDPahsxEMa1Sdu47j8ngV56ETWFXmp2fWhyNOQScnKgTgJeZ5mVZ20RrbRIs4nNxu+RU56hl75ID4WSQkPote9Q2U5pm/QDwcf3m2E0kxZKOgrDr8HK6oOHj9Zqj+tPnj57/qKxvnHgTGkF9oRxh6l4FBJjT2SpPCosAh5qvAwPdmZ8NTtE4a/YGmBQ5yGmZSQHko6SxF59PEvlukQ8Pj9u85hwQhUfWXPmODg+4zIGVYzhD5PaEcKQm2xOj9u/edJohq1wIX7fRLem2XlZXF58u9ruJo3P8dCIMkdNQoFz/SgsaFCBJSkUzupx6bAcQIj7HurIUc3qBY7z/gbnwx5Zqx/mvgi/bujgty5aZ76yhxo7O6yefg/1i8p2x5UhcloRbLQVmpOBk+PyAfSouC1NQbEFb6v3IxBguC/Jnr/gjR3ZXvm4N2Kwpb0X7U7LxnS9XYK/avWUR2Idtsu6rMcE+8i+sO/sOvgUXAU3wY9l6Upw27PJ/lHw8xcwl627</latexit><latexit sha1_base64="vtehmgvKW9+ibfNlf7CzDKgGgWA=">ACM3icbZDPahsxEMa1Sdu47j8ngV56ETWFXmp2fWhyNOQScnKgTgJeZ5mVZ20RrbRIs4nNxu+RU56hl75ID4WSQkPote9Q2U5pm/QDwcf3m2E0kxZKOgrDr8HK6oOHj9Zqj+tPnj57/qKxvnHgTGkF9oRxh6l4FBJjT2SpPCosAh5qvAwPdmZ8NTtE4a/YGmBQ5yGmZSQHko6SxF59PEvlukQ8Pj9u85hwQhUfWXPmODg+4zIGVYzhD5PaEcKQm2xOj9u/edJohq1wIX7fRLem2XlZXF58u9ruJo3P8dCIMkdNQoFz/SgsaFCBJSkUzupx6bAcQIj7HurIUc3qBY7z/gbnwx5Zqx/mvgi/bujgty5aZ76yhxo7O6yefg/1i8p2x5UhcloRbLQVmpOBk+PyAfSouC1NQbEFb6v3IxBguC/Jnr/gjR3ZXvm4N2Kwpb0X7U7LxnS9XYK/avWUR2Idtsu6rMcE+8i+sO/sOvgUXAU3wY9l6Upw27PJ/lHw8xcwl627</latexit><latexit sha1_base64="kqx7jvTquSqPMklaG7enHKPIg5M=">ACM3icbZBNTxsxEIa90BafhDgyMVqVKmXRrs5tByRuCBOVGpIpGxYzTqziYXtmzkGjJf+LCH+GAhHoQlz5D3VCqvLRkSy9ep8ZjedNCyUdheF1sLT86vWbldW3tXfvP3xcq69vHDpTWoFtYZSx3RQcKqmxTZIUdguLkKcKO+nx7ox3TtA6afRPmhTYz2GoZSYFkLeS+n58Nk7k13ES8fjsqMVjwjFVfGjNqePg+JTLGFQxgn9MakcIA26yGT1q/eVJvRE2w3nxlyJaiAZb1EFSv4wHRpQ5ahIKnOtFYUH9CixJoXBai0uHBYhjGLPSw05un41v3nKP3tnwDNj/dPE5+7jiQpy5yZ56jtzoJF7zmbm/1ivpGy7X0ldlIRaPCzKSsXJ8FmAfCAtClITL0BY6f/KxQgsCPIx13wI0fOTX4rDVjMKm9GPqLHzbRHKtin9gXFrHvbIftsQPWZoKdsyv2m90EF8Gv4Da4e2hdChYzm+xJBfd/AOzvqdA=</latexit>

xn x1 = α P

i rfσ(i)(xi)

<latexit sha1_base64="BaEnxBoY2YWZRGgqlgXmNwr453Q=">ACGHicbVA9axtBEJ2Tk1hRvi52mWYTEZCKHdp7MYgcJPKyGBZAp045lZ70qLdvWN3z0gcavIf0qTy/0iTIsa4ded/49VH4Uh5MPB4b4aZeUkuLFB8OBV9p49f7FfVl79frN23f+4NLkxWasi7NRKb7CRomuGJdy61g/VwzlIlgvWR6uvR7V0wbnqkLO8/ZUOJY8ZRTtE6K/a+zWH2ZxeFJhCKfIlMIWNOIoWJQJLGZWT4WGKDNxeNWcybsV8PWsEKZJeEG1Jvfz7cQ0Andi/j0YZLSRTlgo0ZhAGuR2WqC2ngi1qUWFYjnSKYzZwVKFkZliuHluQz04ZkTrpQlK/XpRInSmLlMXKdEOzHb3lL8nzcobHo8LnKC8sUXS9KC0FsRpYpkRHXjFoxdwSp5u5WQieokVqXZc2FEG6/vEsuv7XCoBWeh/X2MaxRhQ/wCRoQwhG04Tt0oAsUfsJv+As3i/vj3fr3a1bK95m5hD+gXf/CLD3oQY=</latexit><latexit sha1_base64="efAvZuLZ/eH5Y9ISiN0hZITw6k=">ACGHicbVA9SwNBEN3zM8avqKXNahCSwnhnYxohYGMlCiYGcuGY2+wli7t7x+6eJBxp/A3a5K/YWChia+e/cZNYaPTBwO9GWbmhQln2rjupzM3v7C4tJxbya+urW9sFra2GzpOFaF1EvNYNUPQlDNJ64YZTpuJoiBCTm/C27Oxf3NHlWaxvDaDhLYFdCWLGAFjpaBw1A/kYT/wTn3gSQ+wr1MRMOxLCDngKMh8zboCSqw8LPUDVg4KRbfiToD/Eu+bFGt7F/cPo0bzMih8+J2YpIJKQzho3fLcxLQzUIYRTod5P9U0AXILXdqyVIKgup1NHhviA6t0cBQrW9LgifpzIgOh9UCEtlOA6elZbyz+57VSE1XbGZNJaqgk0VRyrGJ8Tgl3GKEsMHlgBRzN6KSQ8UEGOzNsQvNmX/5LGcVzK96V6xV0RQ5tIv2UQl56ATV0Dm6RHVE0CN6Qi/o1Rk5z86b8z5tnXO+Z3bQLzgfX1dnokI=</latexit><latexit sha1_base64="efAvZuLZ/eH5Y9ISiN0hZITw6k=">ACGHicbVA9SwNBEN3zM8avqKXNahCSwnhnYxohYGMlCiYGcuGY2+wli7t7x+6eJBxp/A3a5K/YWChia+e/cZNYaPTBwO9GWbmhQln2rjupzM3v7C4tJxbya+urW9sFra2GzpOFaF1EvNYNUPQlDNJ64YZTpuJoiBCTm/C27Oxf3NHlWaxvDaDhLYFdCWLGAFjpaBw1A/kYT/wTn3gSQ+wr1MRMOxLCDngKMh8zboCSqw8LPUDVg4KRbfiToD/Eu+bFGt7F/cPo0bzMih8+J2YpIJKQzho3fLcxLQzUIYRTod5P9U0AXILXdqyVIKgup1NHhviA6t0cBQrW9LgifpzIgOh9UCEtlOA6elZbyz+57VSE1XbGZNJaqgk0VRyrGJ8Tgl3GKEsMHlgBRzN6KSQ8UEGOzNsQvNmX/5LGcVzK96V6xV0RQ5tIv2UQl56ATV0Dm6RHVE0CN6Qi/o1Rk5z86b8z5tnXO+Z3bQLzgfX1dnokI=</latexit><latexit sha1_base64="s9K/xu0KiVtNgstuJvYmBR/fWvg=">ACGHicbVA9SwNBEN2L3/Hr1NJmMQhJYbyzMY0QsLFUMB+QC8fcZi9Zsrt37O5JwpGfYeNfsbFQxNbOf+MmptDEBwOP92aYmRelnGnjeV9OYWV1bX1jc6u4vbO7t+8eHDZ1kilCGyThiWpHoClnkjYM5y2U0VBRJy2ouH1G89UKVZIu/NOKVdAX3JYkbAWCl0z0ehPBuF/lUAPB0ADnQmQoYDCREHId5oFlfQJlVJuVRyCqhW/Kq3gx4mfhzUkJz3IbuZ9BLSCaoNISD1h3fS03B2UY4XRSDJNUyBD6NOpRIE1d189tgEn1qlh+NE2ZIGz9TfEzkIrcisp0CzEAvelPxP6+TmbjWzZlM0Ml+VkUZxybBE9Twj2mKDF8bAkQxeytmAxATE2y6INwV98eZk0L6q+V/Xv/FK9No9jEx2jE1RGPrpEdXSDblEDEfSIntErenOenBfn3fn4aS0485kj9AfO5zc9gJ81</latexit>

⇡ α P

i rfσ(i)(x1)

<latexit sha1_base64="m962x1RphqOQTVsqNVKsq6iudcs=">ACGHicbVCxThtBEJ0DQoxJwJCSZomFZDfOHQ2UlmiokCPF2JLPOs2t9+wVu3ur3T2EdXKTf6Ch4j9oKIitHT5m6xtCsB50khP781oZl6qBbcuDP8Ga+sbHzY/Vraq258+7+zW9vYvbV4Yyro0F7np2iZ4Ip1HXeC9bVhKFPBeunV2dzvXTNjea5+uKlmQ4ljxTNO0XkpqX2LUWuT35AYhZ4giW0hE05ihalAkiVlbPlYoM3Z42bJGomtXrYChcgqyR6IfX24cXPewDoJLXneJTQjLlqEBrB1Go3bBE4zgVbFaNC8s0iscs4GnCiWzw3Lx2IwceWVEstz4Uo4s1NcTJUprpzL1nRLdxL735uL/vEHhstNhyZUuHFN0uSgrBHE5madERtw6sTUE6SG+1sJnaB6nyWVR9C9P7lVXJ53IrCVvQ9qrdPYkKHMBXaEAEJ9CGc+hAFyjcwgM8wa/gLngMfgd/lq1rwcvMF3iD4Pkf+wehLQ=</latexit><latexit sha1_base64="4V8u+dtk6bJM2HMpvjH0m+AM1AQ=">ACGHicbVC5TgMxEPVyE64AJY0hQgpN2KUhZSQaKgQSOaRstJp1vImF7bVsL0q0SsM3QMOv0FCAEC0df4NzFxPGunpvRnNzIsVZ8b6/qc3N7+wuLS8slpYW9/Y3Cpu7zRMmlC6yTlqW7FYChnktYts5y2lKYgYk6b8c3Z2G/eUm1YKq/tUNGOgJ5kCSNgnRQVj0NQSqcDHAJXfcChyUTEcCgh5oCTKA8N6wkos6NReRAFR1Gx5Ff8CfBfEsxIqbZ/cXf/2GhdRsWPsJuSTFBpCQdj2oGvbCcHbRnhdFQIM0MVkBvo0bajEgQ1nXzy2AgfOqWLk1S7khZP1O8TOQhjhiJ2nQJs3/z2xuJ/XjuzSbWTM6kySyWZLkoyjm2KxynhLtOUWD50BIhm7lZM+qCBWJdlwYUQ/H75L2mcVAK/ElwFpVoVTbGC9tABKqMAnaIaOkeXqI4IekBP6AW9eo/es/fmvU9b57zZzC76Ae/jC6F3omk=</latexit><latexit sha1_base64="4V8u+dtk6bJM2HMpvjH0m+AM1AQ=">ACGHicbVC5TgMxEPVyE64AJY0hQgpN2KUhZSQaKgQSOaRstJp1vImF7bVsL0q0SsM3QMOv0FCAEC0df4NzFxPGunpvRnNzIsVZ8b6/qc3N7+wuLS8slpYW9/Y3Cpu7zRMmlC6yTlqW7FYChnktYts5y2lKYgYk6b8c3Z2G/eUm1YKq/tUNGOgJ5kCSNgnRQVj0NQSqcDHAJXfcChyUTEcCgh5oCTKA8N6wkos6NReRAFR1Gx5Ff8CfBfEsxIqbZ/cXf/2GhdRsWPsJuSTFBpCQdj2oGvbCcHbRnhdFQIM0MVkBvo0bajEgQ1nXzy2AgfOqWLk1S7khZP1O8TOQhjhiJ2nQJs3/z2xuJ/XjuzSbWTM6kySyWZLkoyjm2KxynhLtOUWD50BIhm7lZM+qCBWJdlwYUQ/H75L2mcVAK/ElwFpVoVTbGC9tABKqMAnaIaOkeXqI4IekBP6AW9eo/es/fmvU9b57zZzC76Ae/jC6F3omk=</latexit><latexit sha1_base64="Bnw3nw1AyzqMjaxzERVkFMfmGc4=">ACGHicbVA9TwJBEN3zE/ELtbTZSEygwTsbKUlsLDERJOHIZW7Zg427e5vdPQO58DNs/Cs2FhpjS+e/cUEKBV8yct7M5mZFyvOjPX9L29tfWNza7uwU9zd2z84LB0dt02aUJbJOWp7sRgKGeStiyznHaUpiBiTu/jh+uZf/9ItWGpvLNjRXsCBpIljIB1UlS6CEpnY5wCFwNAYcmExHDoYSYA06iPDRsIKDCqpPKAqUans1/w58CoJFqSMFmhGpWnYT0kmqLSEgzHdwFe2l4O2jHA6KYaZoQrIAwxo1EJgpePn9sgs+d0sdJql1Ji+fq74kchDFjEbtOAXZolr2Z+J/XzWxS7+VMqsxSX4WJRnHNsWzlHCfaUosHzsCRDN3KyZD0ECsy7LoQgiWX14l7cta4NeC26DcqC/iKBTdIYqKEBXqIFuUBO1EFP6AW9oXfv2Xv1PrzPn9Y1bzFzgv7Am34Dh5CfXA=</latexit>

= αnrF(x1)

<latexit sha1_base64="w5kMtf5mgOX/Eh3CjCe4n9TihKI=">ACAXicbVDLSgNBEOz1GeNr1UvAy2AQ4iXsejEXISCIxwjmAUkIvZNJMmR2dpmZFWOIF3/FiwdFvPoX3gQ/xsnjoIkFDUVN91dQSy4Np735Swtr6yurac20ptb2zu7t5+RUeJoqxMIxGpWoCaCS5Z2XAjWC1WDMNAsGrQvxj71VumNI/kjRnErBliV/IOp2is1HIz56SBIu4hkaQhMRBILnN3Lf+k5Wa9vDcBWST+jGSLmftvsCi13M9GO6JyKShArWu+15smkNUhlPBRulGolmMtI9dVrdUYsh0czj5YESOrdImnUjZkoZM1N8TQwy1HoSB7QzR9PS8Nxb/8+qJ6RSaQy7jxDBJp4s6iSAmIuM4SJsrRo0YWIJUcXsroT1USI0NLW1D8OdfXiSV07zv5f1rP1swBQpOIQjyIEPZ1CEKyhBGSg8wBO8wKvz6Dw7b87tHXJmc0cwB84Hz/9xZbC</latexit><latexit sha1_base64="dL19A0xdTE9kYoO2ob6c+ou/HW8=">ACAXicbVDLSgNBEJz1mcTXqpeAl8EgxEvY9WIuQlAQjxHMA5Kw9E4myZDZ2WVmVoxLRPBXvHhQJFf/wpvgxzh5HDSxoKGo6qa7y484U9pxvqyl5ZXVtfVUOrOxubW9Y+/uVUYS0IrJOShrPugKGeCVjTnNYjSHwOa35/YuxX7ulUrFQ3OhBRFsBdAXrMALaSJ6dPcN4FEPsMBNAT4HfJm/89xjz845BWcCvEjcGcmVsvf6cfRedmzP5vtkMQBFZpwUKrhOpFuJSA1I5wOM81Y0QhIH7q0YaiAgKpWMvlgiI+M0sadUJoSGk/U3xMJBEoNAt90BqB7at4bi/95jVh3iq2EiSjWVJDpok7MsQ7xOA7cZpISzQeGAJHM3IpJDyQbULmBDc+ZcXSfWk4DoF9rNlYpoihQ6QIcoj1x0ikroCpVRBRH0gJ7RK3qznqwX690aTVuXrNnMPvoD6+MH+zGYPw=</latexit><latexit sha1_base64="dL19A0xdTE9kYoO2ob6c+ou/HW8=">ACAXicbVDLSgNBEJz1mcTXqpeAl8EgxEvY9WIuQlAQjxHMA5Kw9E4myZDZ2WVmVoxLRPBXvHhQJFf/wpvgxzh5HDSxoKGo6qa7y484U9pxvqyl5ZXVtfVUOrOxubW9Y+/uVUYS0IrJOShrPugKGeCVjTnNYjSHwOa35/YuxX7ulUrFQ3OhBRFsBdAXrMALaSJ6dPcN4FEPsMBNAT4HfJm/89xjz845BWcCvEjcGcmVsvf6cfRedmzP5vtkMQBFZpwUKrhOpFuJSA1I5wOM81Y0QhIH7q0YaiAgKpWMvlgiI+M0sadUJoSGk/U3xMJBEoNAt90BqB7at4bi/95jVh3iq2EiSjWVJDpok7MsQ7xOA7cZpISzQeGAJHM3IpJDyQbULmBDc+ZcXSfWk4DoF9rNlYpoihQ6QIcoj1x0ikroCpVRBRH0gJ7RK3qznqwX690aTVuXrNnMPvoD6+MH+zGYPw=</latexit><latexit sha1_base64="K+U7tQWu1FSKiA7KntqZP51Krg8=">ACAXicbVBNS8NAEN3Ur1q/ol4EL4tFqJeSeLEXoSCIxwr2A5oQJtNu3SzCbsbsZR68a948aCIV/+FN/+N2zYHbX0w8Hhvhpl5YcqZ0o7zbRVWVtfWN4qbpa3tnd09e/+gpZJMEtokCU9kJwRFORO0qZnmtJNKCnHIaTscXk39j2ViXiTo9S6sfQFyxiBLSRAvoEnvA0wFgT0BIQd8XkI3LPALjtVZwa8TNyclFGORmB/eb2EZDEVmnBQqus6qfbHIDUjnE5KXqZoCmQIfdo1VEBMlT+efTDBp0bp4SiRpoTGM/X3xBhipUZxaDpj0AO16E3F/7xupqOaP2YizTQVZL4oyjWCZ7GgXtMUqL5yBAgkplbMRmABKJNaCUTgrv48jJpnVdp+reuV6LY+jiI7RCaogF12gOrpBDdREBD2iZ/SK3qwn68V6tz7mrQUrnzlEf2B9/gC4v5U</latexit>
slide-10
SLIDE 10

Iterate coupling

Expected gradient is biased! Expected gradient is approximately unbiased

[4]: Prateek Jain, Dheeraj Nagaraj, and Praneeth Netrapalli. Sgd without replacement: Sharper rates for general smooth convex functions. 2019

  • Same coupling as [4]
  • Assume n = 5: {f1 , f2 , f3 , f4 , f5 }

x1

<latexit sha1_base64="hU9lIzUMPU+1y9fY8ZOBAgrokXI=">AB6nicbZDLSgMxFIbPWC+13qou3QSL4KpM3Oiy4MZlRXuBtpRMeqYNzWSGJCOWoY/gxoUibn0Qn8GF4Noelo6w+Bj/8/h5xzgkQKY3/y1vJra6tb+Q3C1vbO7t7xf2DuolTzbHGYxnrZsAMSqGwZoWV2Ew0siQ2AiGl5O8cYfaiFjd2lGCnYj1lQgFZ9ZN/d2i2W/LI/FVkGOodSJf5/Q4A1W7xo92LeRqhslwyY1rUT2wnY9oKLnFcaKcGE8aHrI8th4pFaDrZdNQxOXFOj4Sxdk9ZMnV/d2QsMmYUBa4yYnZgFrOJ+V/WSm140cmESlKLis8+ClNJbEwme5Oe0MitHDlgXAs3K+EDphm37joFdwS6uPIy1M/K1C/Ta1q+DBTHo7gGE6BwjlU4AqUAMOfXiAJ3j2pPfovXivs9IVb95zCH/kvf0A23eQYw=</latexit><latexit sha1_base64="tKmnOhlmkRX0QAhN9M1NIsWkGJQ=">AB6nicbZC7SgNBFIbPGi8x3qJiZbMYBKuwY6NlwMYyorlAsoTZydlkyOzsMjMrhiWPYGOhiK0v4itYCFY+ik4uhSb+MPDx/+cw5wgEVwbz/t0lnLK6tr+fXCxubW9k5xd6+u41QxrLFYxKoZUI2CS6wZbgQ2E4U0CgQ2gsHFOG/cotI8ljdmKAf0Z7kIWfUWOv6rkM6xZJX9iZyF4HMoFTJfXy/HXxhtVN8b3djlkYoDRNU6xbxEuNnVBnOBI4K7VRjQtmA9rBlUdItZ9NRh25x9bpumGs7JPGnbi/OzIaT2MAlsZUdPX89nY/C9rpSY89zMuk9SgZNOPwlS4JnbHe7tdrpAZMbRAmeJ2Vpf1qaLM2OsU7BHI/MqLUD8tE69Mrkip4sFUeTiEIzgBAmdQgUuoQg0Y9OAeHuHJEc6D8+y8TEuXnFnPvyR8/oDqYORvQ=</latexit><latexit sha1_base64="tKmnOhlmkRX0QAhN9M1NIsWkGJQ=">AB6nicbZC7SgNBFIbPGi8x3qJiZbMYBKuwY6NlwMYyorlAsoTZydlkyOzsMjMrhiWPYGOhiK0v4itYCFY+ik4uhSb+MPDx/+cw5wgEVwbz/t0lnLK6tr+fXCxubW9k5xd6+u41QxrLFYxKoZUI2CS6wZbgQ2E4U0CgQ2gsHFOG/cotI8ljdmKAf0Z7kIWfUWOv6rkM6xZJX9iZyF4HMoFTJfXy/HXxhtVN8b3djlkYoDRNU6xbxEuNnVBnOBI4K7VRjQtmA9rBlUdItZ9NRh25x9bpumGs7JPGnbi/OzIaT2MAlsZUdPX89nY/C9rpSY89zMuk9SgZNOPwlS4JnbHe7tdrpAZMbRAmeJ2Vpf1qaLM2OsU7BHI/MqLUD8tE69Mrkip4sFUeTiEIzgBAmdQgUuoQg0Y9OAeHuHJEc6D8+y8TEuXnFnPvyR8/oDqYORvQ=</latexit><latexit sha1_base64="40IlArGvoEpoZcPz7T28ohAl/vA=">AB6nicbVA9TwJBEJ3DL8Qv1NJmIzGxIrc2WpLYWGIUMIEL2Vv2YMPe3mV3zkgu/AQbC42x9RfZ+W9c4AoFXzLJy3szmZkXpkpa9P1vr7S2vrG5Vd6u7Ozu7R9UD4/aNskMFy2eqMQ8hMwKJbVoUQlHlIjWBwq0QnH1zO/8yiMlYm+x0kqgpgNtYwkZ+iku6c+7Vdrft2fg6wSWpAaFGj2q1+9QcKzWGjkilnbpX6KQc4MSq7EtNLrEgZH7Oh6DqWSxskM9PnZIzpwxIlBhXGslc/T2Rs9jaSRy6zpjhyC57M/E/r5thdBXkUqcZCs0Xi6JMEUzI7G8ykEZwVBNHGDfS3Ur4iBnG0aVTcSHQ5ZdXSfuiTv06vaW1hl/EUYTOIVzoHAJDbiBJrSAwxCe4RXePOW9eO/ex6K15BUzx/AH3ucPBVaNiw=</latexit>

rf3

<latexit sha1_base64="bVdcC6X5ncZLruN6ANlaRVNu2tc=">AB8XicdZDLSgMxFIbPeG3rerSTbAIroZMSy/uCm5cVrAXbEvJpJk2NJMZkoxQhq59ATculOLWt3Hn25i2Cir6Q+Dj/8h5xw/FlwbjN+dtfWNza3tTDa3s7u3f5A/PGrpKFGUNWkItXxiWaCS9Y03AjWiRUjoS9Y259cLvL2HVOaR/LGTGPWD8lI8oBTYqx125PEFwQFg9IgX8BuqVKuVisIu5VS8aJUtlAs4poFz8VLFerZ+EcABqD/FtvGNEkZNJQbTuejg2/ZQow6lgs1wv0SwmdEJGrGtRkpDpfrqceIbOrDNEQaTskwYt3e8dKQm1noa+rQyJGevf2cL8K+smJqj1Uy7jxDBJVx8FiUAmQov10ZArRo2YWiBUcTsromOiCDX2SDl7hK9N0f/QKroedr1r1DHsFIGTuAUzsGDKtThChrQBAoSHuAJnh3tPDpz52VuZ89hzDzmvH67kmE=</latexit><latexit sha1_base64="WhLBfJ3U176yY+7cJVpoM63pXs=">AB8XicdZDLSgMxFIYz9dbW9Wlm2ARXMiQ6dCLu4IblxXsBduhZNJMG5rJDElGKEM3bnwBNy4UcSu+jDufRtNWQUV/CHz8/znknOPHnCmN0JuVWVpeWV3L5vLrG5tb24Wd3ZaKEklok0Q8kh0fK8qZoE3NKedWFIc+py2/fHpLG9fUalYJC70JKZeiIeCBYxgbazLnsA+xzDou/1CEdlupVytViCyK27pxC0bKJVQzYBjo7mK9dzN4OX9+rjRL7z2BhFJQio04ViproNi7aVYakY4neZ7iaIxJmM8pF2DAodUel84ik8NM4ABpE0T2g4d793pDhUahL6pjLEeqR+ZzPzr6yb6KDmpUzEiaCLD4KEg51BGfrwGTlGg+MYCJZGZWSEZYqLNkfLmCF+bwv+hVbIdZDvnTrGOwEJZsA8OwBFwQBXUwRlogCYgQIBbcA8eLGXdWY/W06I0Y327IEfsp4/AP5rlBs=</latexit><latexit sha1_base64="WhLBfJ3U176yY+7cJVpoM63pXs=">AB8XicdZDLSgMxFIYz9dbW9Wlm2ARXMiQ6dCLu4IblxXsBduhZNJMG5rJDElGKEM3bnwBNy4UcSu+jDufRtNWQUV/CHz8/znknOPHnCmN0JuVWVpeWV3L5vLrG5tb24Wd3ZaKEklok0Q8kh0fK8qZoE3NKedWFIc+py2/fHpLG9fUalYJC70JKZeiIeCBYxgbazLnsA+xzDou/1CEdlupVytViCyK27pxC0bKJVQzYBjo7mK9dzN4OX9+rjRL7z2BhFJQio04ViproNi7aVYakY4neZ7iaIxJmM8pF2DAodUel84ik8NM4ABpE0T2g4d793pDhUahL6pjLEeqR+ZzPzr6yb6KDmpUzEiaCLD4KEg51BGfrwGTlGg+MYCJZGZWSEZYqLNkfLmCF+bwv+hVbIdZDvnTrGOwEJZsA8OwBFwQBXUwRlogCYgQIBbcA8eLGXdWY/W06I0Y327IEfsp4/AP5rlBs=</latexit><latexit sha1_base64="Z4D1AjxlyKvNOpS2KoJHzv4SLe4=">AB8XicdVDJSgNBEK1xjXGLevTSGARPQ8+ELN4CXjxGMAsmQ+jp9CRNenqG7h4hPyFw+KePVvPk3dhZBR8UPN6roqpemAquDcYfztr6xubWdm4nv7u3f3BYODpu6SRTlDVpIhLVCYlmgkvWNwI1kVI3EoWDscX839j1Tmify1kxSFsRkKHnEKTFWutJEgqCon6pXyhit1QpV6sVhN1Kyb8slS3xfVyzxHPxAkVYodEvPcGCc1iJg0VROuh1MTIkynAo2y/cyzVJCx2TIupZKEjMdTBcXz9C5VQYoSpQtadBC/T4xJbHWkzi0nTExI/3bm4t/ed3MRLVgymWaGSbpclGUCWQSNH8fDbhi1IiJYQqbm9FdEQUocaGlLchfH2K/ict3/Ww6914xTpexZGDUziDC/CgCnW4hgY0gYKEB3iCZ0c7j86L87psXNWMyfwA87bJzdZkI0=</latexit>

rf5

<latexit sha1_base64="FmrzGBxHwCcs3mHXjSUxBwT3HU=">AB8XicdVDLSgMxFL3js62vqks3wSK4Kkf2O4KblxWsA9sS8mkmTY0kxmSjFBK1/6AGxdKcevfuPNvTFsFT1w4XDOvdx7jx9LYSzG797a+sbm1nYqndnZ3ds/yB4eNU2UaMYbLJKRbvUcCkUb1hJW/HmtPQl7zljy8XfuOayMidWMnMe+FdKhEIBi1TrtKupLioJ+uZ/N4XwV40KlgHC+hMuEdIqVglRUTyeIlcLX0/mANAvZ96w4iloRcWSapMR2CY9ubUm0Fk3yW6SaGx5SN6ZB3HFU05KY3XV48Q2dOGaAg0q6URUv1+8SUhsZMQt91htSOzG9vIf7ldRIbVHpToeLEcsVWi4JEIhuhxftoIDRnVk4coUwLdytiI6opsy6kjAvh61P0P2kW8sQlc01yNQwrpOAETuEcCFxADa6gDg1goOABnuDZM96jN/deVq1r3ufMfyA9/oBiQmSRw=</latexit><latexit sha1_base64="m6uBourYdS4Z/Is3BnQeQodyM=">AB8XicdVDLSgMxFM3UV1tfVZdugkVwISXpA9tdwY3LCvaB7VAyaYNzWSGJCOUoRs3/oAbF4q4FX/GnV+jaugogcuHM65l3v8SLBtUHozUktLa+srqUz2fWNza3t3M5uS4exoqxJQxGqjkc0E1ypuFGsE6kGAk8wdre+HTmt6+Y0jyUF2YSMTcgQ8l9Tomx0mVPEk8Q6Pcr/VweFWoIFatFiAplVMEYW4LpRouQVxAc+TrmZvBy/v1caOfe+0NQhoHTBoqiNZdjCLjJkQZTgWbZnuxZhGhYzJkXUslCZh2k/nFU3holQH0Q2VLGjhXv08kJNB6Eni2MyBmpH97M/Evrxsbv+omXEaxYZIuFvmxgCaEs/fhgCtGjZhYQqji9lZIR0QRamxIWRvC16fwf9IqFrBN5hzn6wgskAb74AcAQxOQB2cgQZoAgokuAX34MHRzp3z6DwtWlPO58we+AHn+QPYkJQB</latexit><latexit sha1_base64="m6uBourYdS4Z/Is3BnQeQodyM=">AB8XicdVDLSgMxFM3UV1tfVZdugkVwISXpA9tdwY3LCvaB7VAyaYNzWSGJCOUoRs3/oAbF4q4FX/GnV+jaugogcuHM65l3v8SLBtUHozUktLa+srqUz2fWNza3t3M5uS4exoqxJQxGqjkc0E1ypuFGsE6kGAk8wdre+HTmt6+Y0jyUF2YSMTcgQ8l9Tomx0mVPEk8Q6Pcr/VweFWoIFatFiAplVMEYW4LpRouQVxAc+TrmZvBy/v1caOfe+0NQhoHTBoqiNZdjCLjJkQZTgWbZnuxZhGhYzJkXUslCZh2k/nFU3holQH0Q2VLGjhXv08kJNB6Eni2MyBmpH97M/Evrxsbv+omXEaxYZIuFvmxgCaEs/fhgCtGjZhYQqji9lZIR0QRamxIWRvC16fwf9IqFrBN5hzn6wgskAb74AcAQxOQB2cgQZoAgokuAX34MHRzp3z6DwtWlPO58we+AHn+QPYkJQB</latexit><latexit sha1_base64="nNW1DOvB6GgzhHAZfIoGCio2iK8=">AB8XicdVDLSgMxFM34rPVdekmWARXJekD213BjcsK9oHtUO6kmTY0kxmSjFBK/8KNC0Xc+jfu/BvTh6CiBy4czrmXe+8JEimMJeTDW1vf2Nzazuxkd/f2Dw5zR8ctE6ea8SaLZaw7ARguheJNK6zknURziALJ28H4au637k2Ila3dpJwP4KhEqFgYJ101MQSMBhv9LP5UmhRkixWsSkUCYVSqkjtFyq0RKmBbJAHq3Q6Ofe4OYpRFXlkwpktJYv0paCuY5LNsLzU8ATaGIe86qiDixp8uLp7hc6cMcBhrV8rihfp9YgqRMZMocJ0R2JH57c3Fv7xuasOqPxUqS1XbLkoTCW2MZ6/jwdCc2blxBFgWrhbMRuBmZdSFkXwten+H/SKhaoS+aG5utkFUcGnaIzdIEoukR1dI0aqIkYUugBPaFnz3iP3ov3umxd81YzJ+gHvLdPEX6Qcw=</latexit>

rf3

<latexit sha1_base64="bVdcC6X5ncZLruN6ANlaRVNu2tc=">AB8XicdZDLSgMxFIbPeG3rerSTbAIroZMSy/uCm5cVrAXbEvJpJk2NJMZkoxQhq59ATculOLWt3Hn25i2Cir6Q+Dj/8h5xw/FlwbjN+dtfWNza3tTDa3s7u3f5A/PGrpKFGUNWkItXxiWaCS9Y03AjWiRUjoS9Y259cLvL2HVOaR/LGTGPWD8lI8oBTYqx125PEFwQFg9IgX8BuqVKuVisIu5VS8aJUtlAs4poFz8VLFerZ+EcABqD/FtvGNEkZNJQbTuejg2/ZQow6lgs1wv0SwmdEJGrGtRkpDpfrqceIbOrDNEQaTskwYt3e8dKQm1noa+rQyJGevf2cL8K+smJqj1Uy7jxDBJVx8FiUAmQov10ZArRo2YWiBUcTsromOiCDX2SDl7hK9N0f/QKroedr1r1DHsFIGTuAUzsGDKtThChrQBAoSHuAJnh3tPDpz52VuZ89hzDzmvH67kmE=</latexit><latexit sha1_base64="WhLBfJ3U176yY+7cJVpoM63pXs=">AB8XicdZDLSgMxFIYz9dbW9Wlm2ARXMiQ6dCLu4IblxXsBduhZNJMG5rJDElGKEM3bnwBNy4UcSu+jDufRtNWQUV/CHz8/znknOPHnCmN0JuVWVpeWV3L5vLrG5tb24Wd3ZaKEklok0Q8kh0fK8qZoE3NKedWFIc+py2/fHpLG9fUalYJC70JKZeiIeCBYxgbazLnsA+xzDou/1CEdlupVytViCyK27pxC0bKJVQzYBjo7mK9dzN4OX9+rjRL7z2BhFJQio04ViproNi7aVYakY4neZ7iaIxJmM8pF2DAodUel84ik8NM4ABpE0T2g4d793pDhUahL6pjLEeqR+ZzPzr6yb6KDmpUzEiaCLD4KEg51BGfrwGTlGg+MYCJZGZWSEZYqLNkfLmCF+bwv+hVbIdZDvnTrGOwEJZsA8OwBFwQBXUwRlogCYgQIBbcA8eLGXdWY/W06I0Y327IEfsp4/AP5rlBs=</latexit><latexit sha1_base64="WhLBfJ3U176yY+7cJVpoM63pXs=">AB8XicdZDLSgMxFIYz9dbW9Wlm2ARXMiQ6dCLu4IblxXsBduhZNJMG5rJDElGKEM3bnwBNy4UcSu+jDufRtNWQUV/CHz8/znknOPHnCmN0JuVWVpeWV3L5vLrG5tb24Wd3ZaKEklok0Q8kh0fK8qZoE3NKedWFIc+py2/fHpLG9fUalYJC70JKZeiIeCBYxgbazLnsA+xzDou/1CEdlupVytViCyK27pxC0bKJVQzYBjo7mK9dzN4OX9+rjRL7z2BhFJQio04ViproNi7aVYakY4neZ7iaIxJmM8pF2DAodUel84ik8NM4ABpE0T2g4d793pDhUahL6pjLEeqR+ZzPzr6yb6KDmpUzEiaCLD4KEg51BGfrwGTlGg+MYCJZGZWSEZYqLNkfLmCF+bwv+hVbIdZDvnTrGOwEJZsA8OwBFwQBXUwRlogCYgQIBbcA8eLGXdWY/W06I0Y327IEfsp4/AP5rlBs=</latexit><latexit sha1_base64="Z4D1AjxlyKvNOpS2KoJHzv4SLe4=">AB8XicdVDJSgNBEK1xjXGLevTSGARPQ8+ELN4CXjxGMAsmQ+jp9CRNenqG7h4hPyFw+KePVvPk3dhZBR8UPN6roqpemAquDcYfztr6xubWdm4nv7u3f3BYODpu6SRTlDVpIhLVCYlmgkvWNwI1kVI3EoWDscX839j1Tmify1kxSFsRkKHnEKTFWutJEgqCon6pXyhit1QpV6sVhN1Kyb8slS3xfVyzxHPxAkVYodEvPcGCc1iJg0VROuh1MTIkynAo2y/cyzVJCx2TIupZKEjMdTBcXz9C5VQYoSpQtadBC/T4xJbHWkzi0nTExI/3bm4t/ed3MRLVgymWaGSbpclGUCWQSNH8fDbhi1IiJYQqbm9FdEQUocaGlLchfH2K/ict3/Ww6914xTpexZGDUziDC/CgCnW4hgY0gYKEB3iCZ0c7j86L87psXNWMyfwA87bJzdZkI0=</latexit>

rf4

<latexit sha1_base64="bslIi/MsCq8XuN6ykcwSk4DzE0w=">AB8XicbVA9SwNBEJ2LX0n8ilraLAbB6tgTxdgFbCwjmA9MjrC3t5cs2ds7dveEcKT2D9hYKMHWf2Pnv3GTWGjig4HezPMmwlSwbXB+MsprK1vbG4VS+Xtnd29/crBYUsnmaKsSRORqE5ANBNcsqbhRrBOqhiJA8Hawehm5rcfmdI8kfdmnDI/JgPJI06JsdJDT5JAEBT1L/qVKnava5feRQ1hF8+BvGVSrZewikANPqVz16Y0Cxm0lBtO56ODV+TpThVLBJuZdplhI6IgPWtVSmGk/nyeoFOrhChKlC1p0Fz9PZGTWOtxHNjOmJihXvZm4n9eNzNRzc+5TDPDJF0sijKBTIJm56OQK0aNGFtCqOI2K6JDog19kl+4SVk1dJ69z1sOvdedU6hgWKcAwncAYeXEdbqEBTaAg4Rle4c3Rzoszd4XrQXnZ+YI/sD5+AZusJI0</latexit><latexit sha1_base64="tPZKkP/xON1to4uOx4smWlPE7UM=">AB8XicbVA9SwNBEN2LX0n8ilraLAbBQo47iRi7gI1lBPOByRHm9vaSJXt7x+6eEI40Nv4BGwtFbMU/Y+ev0U1ioYkPBh7vzTBvxk84U9pxPq3c0vLK6lq+UFzf2NzaLu3sNlWcSkIbJOaxbPugKGeCNjTnLYTSHyOW35w4uJ37qlUrFYXOtRQr0I+oKFjIA20k1XgM8Bh71Kr1R27PqVupYsd2psDuPCnXCvfB+9fdcb1X+ugGMUkjKjThoFTHdRLtZSA1I5yOi91U0QTIEPq0Y6iAiCovmyYe40OjBDiMpSmh8VT9PZFBpNQo8k1nBHqg5r2J+J/XSXVY9TImklRTQWaLwpRjHePJ+ThgkhLNR4YAkcxkxWQAEog2TyqaJycvEiaJ7br2O6VW645aIY82kcH6Ai56AzV0CWqowYiSKAH9ISeLWU9Wi/W6w1Z/3M7KE/sN6+Ab43k+4=</latexit><latexit sha1_base64="tPZKkP/xON1to4uOx4smWlPE7UM=">AB8XicbVA9SwNBEN2LX0n8ilraLAbBQo47iRi7gI1lBPOByRHm9vaSJXt7x+6eEI40Nv4BGwtFbMU/Y+ev0U1ioYkPBh7vzTBvxk84U9pxPq3c0vLK6lq+UFzf2NzaLu3sNlWcSkIbJOaxbPugKGeCNjTnLYTSHyOW35w4uJ37qlUrFYXOtRQr0I+oKFjIA20k1XgM8Bh71Kr1R27PqVupYsd2psDuPCnXCvfB+9fdcb1X+ugGMUkjKjThoFTHdRLtZSA1I5yOi91U0QTIEPq0Y6iAiCovmyYe40OjBDiMpSmh8VT9PZFBpNQo8k1nBHqg5r2J+J/XSXVY9TImklRTQWaLwpRjHePJ+ThgkhLNR4YAkcxkxWQAEog2TyqaJycvEiaJ7br2O6VW645aIY82kcH6Ai56AzV0CWqowYiSKAH9ISeLWU9Wi/W6w1Z/3M7KE/sN6+Ab43k+4=</latexit><latexit sha1_base64="c6et+rsDGehnULb4QgXHwn3Q4=">AB8XicbVDLSgNBEOyNrxhfUY9eBoPgadkVxXgLePEYwTwWULvZDYZMju7zMwKYclfePGgiFf/xpt/4+Rx0MSChqKqm+6uMBVcG8/7dgpr6xubW8Xt0s7u3v5B+fCoqZNMUdagiUhUO0TNBJesYbgRrJ0qhnEoWCsc3U791hNTmifywYxTFsQ4kDziFI2VHrsSQ4Ek6l32yhXPvale+ZdV4rneDMRfJhVYoN4rf3X7Cc1iJg0VqHXH91IT5KgMp4JNSt1MsxTpCAesY6nEmOkgn108IWdW6ZMoUbakITP190SOsdbjOLSdMZqhXvam4n9eJzNRNci5TDPDJ0vijJBTEKm75M+V4waMbYEqeL2VkKHqJAaG1LJhrDy8ipXri+5/r3fqXmLeIowgmcwjn4cA01uIM6NIChGd4hTdHOy/Ou/Mxby04i5lj+APn8wf3FpBg</latexit>

rf4

<latexit sha1_base64="bslIi/MsCq8XuN6ykcwSk4DzE0w=">AB8XicbVA9SwNBEJ2LX0n8ilraLAbB6tgTxdgFbCwjmA9MjrC3t5cs2ds7dveEcKT2D9hYKMHWf2Pnv3GTWGjig4HezPMmwlSwbXB+MsprK1vbG4VS+Xtnd29/crBYUsnmaKsSRORqE5ANBNcsqbhRrBOqhiJA8Hawehm5rcfmdI8kfdmnDI/JgPJI06JsdJDT5JAEBT1L/qVKnava5feRQ1hF8+BvGVSrZewikANPqVz16Y0Cxm0lBtO56ODV+TpThVLBJuZdplhI6IgPWtVSmGk/nyeoFOrhChKlC1p0Fz9PZGTWOtxHNjOmJihXvZm4n9eNzNRzc+5TDPDJF0sijKBTIJm56OQK0aNGFtCqOI2K6JDog19kl+4SVk1dJ69z1sOvdedU6hgWKcAwncAYeXEdbqEBTaAg4Rle4c3Rzoszd4XrQXnZ+YI/sD5+AZusJI0</latexit><latexit sha1_base64="tPZKkP/xON1to4uOx4smWlPE7UM=">AB8XicbVA9SwNBEN2LX0n8ilraLAbBQo47iRi7gI1lBPOByRHm9vaSJXt7x+6eEI40Nv4BGwtFbMU/Y+ev0U1ioYkPBh7vzTBvxk84U9pxPq3c0vLK6lq+UFzf2NzaLu3sNlWcSkIbJOaxbPugKGeCNjTnLYTSHyOW35w4uJ37qlUrFYXOtRQr0I+oKFjIA20k1XgM8Bh71Kr1R27PqVupYsd2psDuPCnXCvfB+9fdcb1X+ugGMUkjKjThoFTHdRLtZSA1I5yOi91U0QTIEPq0Y6iAiCovmyYe40OjBDiMpSmh8VT9PZFBpNQo8k1nBHqg5r2J+J/XSXVY9TImklRTQWaLwpRjHePJ+ThgkhLNR4YAkcxkxWQAEog2TyqaJycvEiaJ7br2O6VW645aIY82kcH6Ai56AzV0CWqowYiSKAH9ISeLWU9Wi/W6w1Z/3M7KE/sN6+Ab43k+4=</latexit><latexit sha1_base64="tPZKkP/xON1to4uOx4smWlPE7UM=">AB8XicbVA9SwNBEN2LX0n8ilraLAbBQo47iRi7gI1lBPOByRHm9vaSJXt7x+6eEI40Nv4BGwtFbMU/Y+ev0U1ioYkPBh7vzTBvxk84U9pxPq3c0vLK6lq+UFzf2NzaLu3sNlWcSkIbJOaxbPugKGeCNjTnLYTSHyOW35w4uJ37qlUrFYXOtRQr0I+oKFjIA20k1XgM8Bh71Kr1R27PqVupYsd2psDuPCnXCvfB+9fdcb1X+ugGMUkjKjThoFTHdRLtZSA1I5yOi91U0QTIEPq0Y6iAiCovmyYe40OjBDiMpSmh8VT9PZFBpNQo8k1nBHqg5r2J+J/XSXVY9TImklRTQWaLwpRjHePJ+ThgkhLNR4YAkcxkxWQAEog2TyqaJycvEiaJ7br2O6VW645aIY82kcH6Ai56AzV0CWqowYiSKAH9ISeLWU9Wi/W6w1Z/3M7KE/sN6+Ab43k+4=</latexit><latexit sha1_base64="c6et+rsDGehnULb4QgXHwn3Q4=">AB8XicbVDLSgNBEOyNrxhfUY9eBoPgadkVxXgLePEYwTwWULvZDYZMju7zMwKYclfePGgiFf/xpt/4+Rx0MSChqKqm+6uMBVcG8/7dgpr6xubW8Xt0s7u3v5B+fCoqZNMUdagiUhUO0TNBJesYbgRrJ0qhnEoWCsc3U791hNTmifywYxTFsQ4kDziFI2VHrsSQ4Ek6l32yhXPvale+ZdV4rneDMRfJhVYoN4rf3X7Cc1iJg0VqHXH91IT5KgMp4JNSt1MsxTpCAesY6nEmOkgn108IWdW6ZMoUbakITP190SOsdbjOLSdMZqhXvam4n9eJzNRNci5TDPDJ0vijJBTEKm75M+V4waMbYEqeL2VkKHqJAaG1LJhrDy8ipXri+5/r3fqXmLeIowgmcwjn4cA01uIM6NIChGd4hTdHOy/Ou/Mxby04i5lj+APn8wf3FpBg</latexit>

rf5

<latexit sha1_base64="FmrzGBxHwCcs3mHXjSUxBwT3HU=">AB8XicdVDLSgMxFL3js62vqks3wSK4Kkf2O4KblxWsA9sS8mkmTY0kxmSjFBK1/6AGxdKcevfuPNvTFsFT1w4XDOvdx7jx9LYSzG797a+sbm1nYqndnZ3ds/yB4eNU2UaMYbLJKRbvUcCkUb1hJW/HmtPQl7zljy8XfuOayMidWMnMe+FdKhEIBi1TrtKupLioJ+uZ/N4XwV40KlgHC+hMuEdIqVglRUTyeIlcLX0/mANAvZ96w4iloRcWSapMR2CY9ubUm0Fk3yW6SaGx5SN6ZB3HFU05KY3XV48Q2dOGaAg0q6URUv1+8SUhsZMQt91htSOzG9vIf7ldRIbVHpToeLEcsVWi4JEIhuhxftoIDRnVk4coUwLdytiI6opsy6kjAvh61P0P2kW8sQlc01yNQwrpOAETuEcCFxADa6gDg1goOABnuDZM96jN/deVq1r3ufMfyA9/oBiQmSRw=</latexit><latexit sha1_base64="m6uBourYdS4Z/Is3BnQeQodyM=">AB8XicdVDLSgMxFM3UV1tfVZdugkVwISXpA9tdwY3LCvaB7VAyaYNzWSGJCOUoRs3/oAbF4q4FX/GnV+jaugogcuHM65l3v8SLBtUHozUktLa+srqUz2fWNza3t3M5uS4exoqxJQxGqjkc0E1ypuFGsE6kGAk8wdre+HTmt6+Y0jyUF2YSMTcgQ8l9Tomx0mVPEk8Q6Pcr/VweFWoIFatFiAplVMEYW4LpRouQVxAc+TrmZvBy/v1caOfe+0NQhoHTBoqiNZdjCLjJkQZTgWbZnuxZhGhYzJkXUslCZh2k/nFU3holQH0Q2VLGjhXv08kJNB6Eni2MyBmpH97M/Evrxsbv+omXEaxYZIuFvmxgCaEs/fhgCtGjZhYQqji9lZIR0QRamxIWRvC16fwf9IqFrBN5hzn6wgskAb74AcAQxOQB2cgQZoAgokuAX34MHRzp3z6DwtWlPO58we+AHn+QPYkJQB</latexit><latexit sha1_base64="m6uBourYdS4Z/Is3BnQeQodyM=">AB8XicdVDLSgMxFM3UV1tfVZdugkVwISXpA9tdwY3LCvaB7VAyaYNzWSGJCOUoRs3/oAbF4q4FX/GnV+jaugogcuHM65l3v8SLBtUHozUktLa+srqUz2fWNza3t3M5uS4exoqxJQxGqjkc0E1ypuFGsE6kGAk8wdre+HTmt6+Y0jyUF2YSMTcgQ8l9Tomx0mVPEk8Q6Pcr/VweFWoIFatFiAplVMEYW4LpRouQVxAc+TrmZvBy/v1caOfe+0NQhoHTBoqiNZdjCLjJkQZTgWbZnuxZhGhYzJkXUslCZh2k/nFU3holQH0Q2VLGjhXv08kJNB6Eni2MyBmpH97M/Evrxsbv+omXEaxYZIuFvmxgCaEs/fhgCtGjZhYQqji9lZIR0QRamxIWRvC16fwf9IqFrBN5hzn6wgskAb74AcAQxOQB2cgQZoAgokuAX34MHRzp3z6DwtWlPO58we+AHn+QPYkJQB</latexit><latexit sha1_base64="nNW1DOvB6GgzhHAZfIoGCio2iK8=">AB8XicdVDLSgMxFM34rPVdekmWARXJekD213BjcsK9oHtUO6kmTY0kxmSjFBK/8KNC0Xc+jfu/BvTh6CiBy4czrmXe+8JEimMJeTDW1vf2Nzazuxkd/f2Dw5zR8ctE6ea8SaLZaw7ARguheJNK6zknURziALJ28H4au637k2Ila3dpJwP4KhEqFgYJ101MQSMBhv9LP5UmhRkixWsSkUCYVSqkjtFyq0RKmBbJAHq3Q6Ofe4OYpRFXlkwpktJYv0paCuY5LNsLzU8ATaGIe86qiDixp8uLp7hc6cMcBhrV8rihfp9YgqRMZMocJ0R2JH57c3Fv7xuasOqPxUqS1XbLkoTCW2MZ6/jwdCc2blxBFgWrhbMRuBmZdSFkXwten+H/SKhaoS+aG5utkFUcGnaIzdIEoukR1dI0aqIkYUugBPaFnz3iP3ov3umxd81YzJ+gHvLdPEX6Qcw=</latexit>

rf2

<latexit sha1_base64="O71XxrES1HvcC7IVcrYqbJ5a+Q=">AB8XicdZDLSgMxFIbPeG3rerSTbAIrkqmIm13BTcuK9gLtkPJpJk2NJMZkoxQhq59ATculOLWt3Hn25hpK6joD4Gf7z+HnHP8WHBtMP5w1tY3Nre2c/nCzu7e/kHx8Kito0R1qKRiFTXJ5oJLlnLcCNYN1aMhL5gHX9yleWde6Y0j+StmcbMC8lI8oBTYiy60viC4KCQWVQLOFyvXZ5UXcRLuOFMlPBuFpD7oqUGvmH4RwAmoPie38Y0SRk0lBtO65ODZeSpThVLBZoZ9oFhM6ISPWs1aSkGkvXUw8Q2eWDFEQKfukQv6vSMlodbT0LeVITFj/TvL4F9ZLzFBzUu5jBPDJF1+FCQCmQhl6MhV4waMbWGUMXtrIiOiSLU2CMV7BG+NkX/m3al7OKye+OWGhiWysEJnMI5uFCFBlxDE1pAQcIjPMOLo50nZ+68LkvXnFXPMfyQ8/YJgimSQg=</latexit><latexit sha1_base64="5QPjwHo1yNUcMIdKtrCtx5h1/o=">AB8XicdZDLSgMxFIYz9dbW9Wlm2ARXMiQqUjbXcGNywr2gu1QMplMG5rJDElGKEM3bnwBNy4UcSu+jDufRjNtBRX9IfDz/eQc4Xc6Y0Qu9Wbml5ZXUtXyiub2xubZd2dtsqSiShLRLxSHY9rChngrY05x2Y0lx6Ha8cZnWd65plKxSFzqSUzdEA8FCxjB2qCrvsAexzAYVAalMrLrtdOTugORjWbKTAWhag06C1JuFG7914+b4+ag9Nb3I5KEVGjCsVI9B8XaTbHUjHA6LfYTRWNMxnhIe8YKHFLlprOJp/DQEB8GkTRPaDij3ztSHCo1CT1TGWI9Ur+zDP6V9RId1NyUiTjRVJD5R0HCoY5gtj70maRE84kxmEhmZoVkhCUm2hypaI7wtSn837QrtoNs58IpNxCYKw/2wQE4Ag6ogY4B03QAgQIcAcewKOlrHvryXqel+asRc8e+CHr5RPRsJP8</latexit><latexit sha1_base64="5QPjwHo1yNUcMIdKtrCtx5h1/o=">AB8XicdZDLSgMxFIYz9dbW9Wlm2ARXMiQqUjbXcGNywr2gu1QMplMG5rJDElGKEM3bnwBNy4UcSu+jDufRjNtBRX9IfDz/eQc4Xc6Y0Qu9Wbml5ZXUtXyiub2xubZd2dtsqSiShLRLxSHY9rChngrY05x2Y0lx6Ha8cZnWd65plKxSFzqSUzdEA8FCxjB2qCrvsAexzAYVAalMrLrtdOTugORjWbKTAWhag06C1JuFG7914+b4+ag9Nb3I5KEVGjCsVI9B8XaTbHUjHA6LfYTRWNMxnhIe8YKHFLlprOJp/DQEB8GkTRPaDij3ztSHCo1CT1TGWI9Ur+zDP6V9RId1NyUiTjRVJD5R0HCoY5gtj70maRE84kxmEhmZoVkhCUm2hypaI7wtSn837QrtoNs58IpNxCYKw/2wQE4Ag6ogY4B03QAgQIcAcewKOlrHvryXqel+asRc8e+CHr5RPRsJP8</latexit><latexit sha1_base64="Fz/zsKoRuXYBt3wY20mXw5hetg=">AB8XicdVBNSwMxEJ2tX7V+VT16CRbB05KtSNtbwYvHCvYD26Vk02wbms0uSVYopf/CiwdFvPpvPlvzLYrqOiDgcd7M8zMCxLBtcH4wymsrW9sbhW3Szu7e/sH5cOjo5TRVmbxiJWvYBoJrhkbcONYL1EMRIFgnWD6VXmd+Z0jyWt2aWMD8iY8lDTomx0t1AkAQFA6rw3IFu4365UXDQ9jFS2SkinGtjrxcqUCO1rD8PhjFNI2YNFQrfseTow/J8pwKtiNEg1SwidkjHrWypJxLQ/X168QGdWGaEwVrakQUv1+8ScRFrPosB2RsRM9G8vE/y+qkJ6/6cyQ1TNLVojAVyMQoex+NuGLUiJklhCpub0V0QhShxoZUsiF8fYr+J52q62HXu/EqTZzHUYQTOIVz8KAGTbiGFrSBgoQHeIJnRzuPzovzumotOPnMfyA8/YJCp6Qbg=</latexit>

rf1

<latexit sha1_base64="rTahWIFj4vpux9DzMKcFtQUmw=">AB8XicbZC7SgNBFIbPxluMt6ilzWAQrMKOjZYBG8sI5oLJEs5OZpMhs7PLzKwQltS+gI2FEmx9Gzvfxsml0MQfBj7+/xzmnBOmUhjr+9eYWNza3unuFva2z84PCofnzRNkmnGyRiW6HaLgUijesJK3U80xDiVvhaPbWd564tqIRD3YcqDGAdKRIKhdZjV2EokUQ92itX/Ko/F1kHuoRKbfe5PwWAeq/81e0nLIu5skyiMR3qpzbIUVvBJ+UupnhKbIRDnjHocKYmyCfTzwhF87pkyjR7ilL5u7vjhxjY8Zx6CpjtEOzms3M/7JOZqObIBcqzSxXbPFRlEliEzJbn/SF5szKsQNkWrhZCRuiRmbdkUruCHR15XVoXlWpX6X3tFLzYaEinME5XAKFa6jBHdShAQwUvMAbvHvGe/Wm3seitOAte07hj7zPHzTJkg0=</latexit><latexit sha1_base64="4+p9tVQPevYSvBbGHQTtGyWSgds=">AB8XicbZC7SgNBFIbPxlsSb1FLm8EgWEjYtdEyYGMZwVwCeHs7GwyZHZ2mZkVwpLGxhewsVDEVnwZO59GJ5dCE38Y+Pj/c5hzjp8Iro3rfjm5ldW19Y18obi5tb2zW9rb+g4VZTVaSxi1fJRM8ElqxtuBGslimHkC9b0h5eTvHnHlOaxvDGjhHUj7EsecorGWrcdib5AEva8XqnsVtypyDJ4cyhXCw/Bx/f9a1X+uwEMU0jJg0VqHXbcxPTzVAZTgUbFzupZgnSIfZ26LEiOluNp14TI6tE5AwVvZJQ6bu74MI61HkW8rIzQDvZhNzP+ydmrCi27GZIaJunsozAVxMRksj4JuGLUiJEFpIrbWQkdoEJq7JGK9gje4srL0DireG7Fu/bKVRdmysMhHMEJeHAOVbiCGtSBgoRHeIYXRztPzqvzNivNOfOeA/gj5/0HhFCTxw=</latexit><latexit sha1_base64="4+p9tVQPevYSvBbGHQTtGyWSgds=">AB8XicbZC7SgNBFIbPxlsSb1FLm8EgWEjYtdEyYGMZwVwCeHs7GwyZHZ2mZkVwpLGxhewsVDEVnwZO59GJ5dCE38Y+Pj/c5hzjp8Iro3rfjm5ldW19Y18obi5tb2zW9rb+g4VZTVaSxi1fJRM8ElqxtuBGslimHkC9b0h5eTvHnHlOaxvDGjhHUj7EsecorGWrcdib5AEva8XqnsVtypyDJ4cyhXCw/Bx/f9a1X+uwEMU0jJg0VqHXbcxPTzVAZTgUbFzupZgnSIfZ26LEiOluNp14TI6tE5AwVvZJQ6bu74MI61HkW8rIzQDvZhNzP+ydmrCi27GZIaJunsozAVxMRksj4JuGLUiJEFpIrbWQkdoEJq7JGK9gje4srL0DireG7Fu/bKVRdmysMhHMEJeHAOVbiCGtSBgoRHeIYXRztPzqvzNivNOfOeA/gj5/0HhFCTxw=</latexit><latexit sha1_base64="8eg7QxhxKCaLa/mqL4PcEPoWhA=">AB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFW5tAzYWEYwH5gcYW6zlyzZ2zt294Rw5F/YWChi67+x89+4Sa7QxAcDj/dmJkXplIY6/vfXmljc2t7p7xb2ds/ODyqHp+0TZJpxlskYnuhmi4FIq3rLCSd1PNMQ4l74ST27nfeLaiEQ92GnKgxhHSkSCoXSY19hKJFEAzqo1vy6vwBZJ7QgNSjQHFS/+sOEZTFXlk0pkf91AY5aiuY5LNKPzM8RTbBEe85qjDmJsgXF8/IhVOGJEq0K2XJQv09kWNszDQOXWeMdmxWvbn4n9fLbHQT5EKlmeWKLRdFmSQ2IfP3yVBozqycOoJMC3crYWPUyKwLqeJCoKsvr5P2VZ36dXpPaw2/iKMZ3AOl0DhGhpwB01oAQMFz/AKb57xXrx372PZWvKmVP4A+/zB70vkDk=</latexit>

rf1

<latexit sha1_base64="rTahWIFj4vpux9DzMKcFtQUmw=">AB8XicbZC7SgNBFIbPxluMt6ilzWAQrMKOjZYBG8sI5oLJEs5OZpMhs7PLzKwQltS+gI2FEmx9Gzvfxsml0MQfBj7+/xzmnBOmUhjr+9eYWNza3unuFva2z84PCofnzRNkmnGyRiW6HaLgUijesJK3U80xDiVvhaPbWd564tqIRD3YcqDGAdKRIKhdZjV2EokUQ92itX/Ko/F1kHuoRKbfe5PwWAeq/81e0nLIu5skyiMR3qpzbIUVvBJ+UupnhKbIRDnjHocKYmyCfTzwhF87pkyjR7ilL5u7vjhxjY8Zx6CpjtEOzms3M/7JOZqObIBcqzSxXbPFRlEliEzJbn/SF5szKsQNkWrhZCRuiRmbdkUruCHR15XVoXlWpX6X3tFLzYaEinME5XAKFa6jBHdShAQwUvMAbvHvGe/Wm3seitOAte07hj7zPHzTJkg0=</latexit><latexit sha1_base64="4+p9tVQPevYSvBbGHQTtGyWSgds=">AB8XicbZC7SgNBFIbPxlsSb1FLm8EgWEjYtdEyYGMZwVwCeHs7GwyZHZ2mZkVwpLGxhewsVDEVnwZO59GJ5dCE38Y+Pj/c5hzjp8Iro3rfjm5ldW19Y18obi5tb2zW9rb+g4VZTVaSxi1fJRM8ElqxtuBGslimHkC9b0h5eTvHnHlOaxvDGjhHUj7EsecorGWrcdib5AEva8XqnsVtypyDJ4cyhXCw/Bx/f9a1X+uwEMU0jJg0VqHXbcxPTzVAZTgUbFzupZgnSIfZ26LEiOluNp14TI6tE5AwVvZJQ6bu74MI61HkW8rIzQDvZhNzP+ydmrCi27GZIaJunsozAVxMRksj4JuGLUiJEFpIrbWQkdoEJq7JGK9gje4srL0DireG7Fu/bKVRdmysMhHMEJeHAOVbiCGtSBgoRHeIYXRztPzqvzNivNOfOeA/gj5/0HhFCTxw=</latexit><latexit sha1_base64="4+p9tVQPevYSvBbGHQTtGyWSgds=">AB8XicbZC7SgNBFIbPxlsSb1FLm8EgWEjYtdEyYGMZwVwCeHs7GwyZHZ2mZkVwpLGxhewsVDEVnwZO59GJ5dCE38Y+Pj/c5hzjp8Iro3rfjm5ldW19Y18obi5tb2zW9rb+g4VZTVaSxi1fJRM8ElqxtuBGslimHkC9b0h5eTvHnHlOaxvDGjhHUj7EsecorGWrcdib5AEva8XqnsVtypyDJ4cyhXCw/Bx/f9a1X+uwEMU0jJg0VqHXbcxPTzVAZTgUbFzupZgnSIfZ26LEiOluNp14TI6tE5AwVvZJQ6bu74MI61HkW8rIzQDvZhNzP+ydmrCi27GZIaJunsozAVxMRksj4JuGLUiJEFpIrbWQkdoEJq7JGK9gje4srL0DireG7Fu/bKVRdmysMhHMEJeHAOVbiCGtSBgoRHeIYXRztPzqvzNivNOfOeA/gj5/0HhFCTxw=</latexit><latexit sha1_base64="8eg7QxhxKCaLa/mqL4PcEPoWhA=">AB8XicbVA9SwNBEJ2LXzF+RS1tFoNgFW5tAzYWEYwH5gcYW6zlyzZ2zt294Rw5F/YWChi67+x89+4Sa7QxAcDj/dmJkXplIY6/vfXmljc2t7p7xb2ds/ODyqHp+0TZJpxlskYnuhmi4FIq3rLCSd1PNMQ4l74ST27nfeLaiEQ92GnKgxhHSkSCoXSY19hKJFEAzqo1vy6vwBZJ7QgNSjQHFS/+sOEZTFXlk0pkf91AY5aiuY5LNKPzM8RTbBEe85qjDmJsgXF8/IhVOGJEq0K2XJQv09kWNszDQOXWeMdmxWvbn4n9fLbHQT5EKlmeWKLRdFmSQ2IfP3yVBozqycOoJMC3crYWPUyKwLqeJCoKsvr5P2VZ36dXpPaw2/iKMZ3AOl0DhGhpwB01oAQMFz/AKb57xXrx372PZWvKmVP4A+/zB70vkDk=</latexit>
slide-11
SLIDE 11

Lower bound

slide-12
SLIDE 12

Lower bound - Function

  • Need non-Lipschitz Hessian : Piece-wise quadratic!

F x = 1 n &

!"# $ %

f& x + &

!"# $ %

f' x where, f& x = )x% + x x ≥ 0 Rx% + x x < 0 and f' x = )x% − x x ≥ 0 Rx% − x x < 0

  • Hessian discontinuous at 0, minimizer at 0.

[3]: Jeffery Z HaoChen and Suvrit Sra. Random shuffling beats sgd after finite epochs. 2018

F(x) = 1 n @

n/2

X

i=1

fa(x) +

n/2

X

i=1

fb(x) 1 A

<latexit sha1_base64="uXsAmFQl76d40yKOkgBCm8Fgl04=">ACOXicbVBNSwMxFMz6WetX1aOXYBEqQt0UQS+CIjHClaFbl2yabYNZrNL8lYsy/4tL/4Lb4IXD4p49Q+YbXtQ60BgmJnHy5sgkcKA6z47U9Mzs3PzpYXy4tLymplbf3SxKlmvMViGevrgBouheItECD5daI5jQLJr4Lbk8K/uPaiFhdwCDhnYj2lAgFo2Alv9I8rd3vHmhpiwjeaZy7EkeQs0zaeRn4ojkN5na+ShT7FN4l086QSF42nR68NO2a9U3bo7BJ4kZEyqaIymX3nyujFLI6ASWpMm7gJdDKqQTDJ87KXGp5Qdkt7vG2pohE3nWx4eY63rdLFYaztU4CH6s+JjEbGDKLAJiMKfPXK8T/vHYK4WEnEypJgSs2WhSmEkOMixpxV2jOQA4soUwL+1fM+tS2CLbsogTy9+RJctmoE7dOzverx2RcRwltoi1UQwQdoGN0hpqohRh6QC/oDb07j86r8+F8jqJTznhmA/2C8/UN+S+rGw=</latexit><latexit sha1_base64="uXsAmFQl76d40yKOkgBCm8Fgl04=">ACOXicbVBNSwMxFMz6WetX1aOXYBEqQt0UQS+CIjHClaFbl2yabYNZrNL8lYsy/4tL/4Lb4IXD4p49Q+YbXtQ60BgmJnHy5sgkcKA6z47U9Mzs3PzpYXy4tLymplbf3SxKlmvMViGevrgBouheItECD5daI5jQLJr4Lbk8K/uPaiFhdwCDhnYj2lAgFo2Alv9I8rd3vHmhpiwjeaZy7EkeQs0zaeRn4ojkN5na+ShT7FN4l086QSF42nR68NO2a9U3bo7BJ4kZEyqaIymX3nyujFLI6ASWpMm7gJdDKqQTDJ87KXGp5Qdkt7vG2pohE3nWx4eY63rdLFYaztU4CH6s+JjEbGDKLAJiMKfPXK8T/vHYK4WEnEypJgSs2WhSmEkOMixpxV2jOQA4soUwL+1fM+tS2CLbsogTy9+RJctmoE7dOzverx2RcRwltoi1UQwQdoGN0hpqohRh6QC/oDb07j86r8+F8jqJTznhmA/2C8/UN+S+rGw=</latexit><latexit sha1_base64="uXsAmFQl76d40yKOkgBCm8Fgl04=">ACOXicbVBNSwMxFMz6WetX1aOXYBEqQt0UQS+CIjHClaFbl2yabYNZrNL8lYsy/4tL/4Lb4IXD4p49Q+YbXtQ60BgmJnHy5sgkcKA6z47U9Mzs3PzpYXy4tLymplbf3SxKlmvMViGevrgBouheItECD5daI5jQLJr4Lbk8K/uPaiFhdwCDhnYj2lAgFo2Alv9I8rd3vHmhpiwjeaZy7EkeQs0zaeRn4ojkN5na+ShT7FN4l086QSF42nR68NO2a9U3bo7BJ4kZEyqaIymX3nyujFLI6ASWpMm7gJdDKqQTDJ87KXGp5Qdkt7vG2pohE3nWx4eY63rdLFYaztU4CH6s+JjEbGDKLAJiMKfPXK8T/vHYK4WEnEypJgSs2WhSmEkOMixpxV2jOQA4soUwL+1fM+tS2CLbsogTy9+RJctmoE7dOzverx2RcRwltoi1UQwQdoGN0hpqohRh6QC/oDb07j86r8+F8jqJTznhmA/2C8/UN+S+rGw=</latexit><latexit sha1_base64="uXsAmFQl76d40yKOkgBCm8Fgl04=">ACOXicbVBNSwMxFMz6WetX1aOXYBEqQt0UQS+CIjHClaFbl2yabYNZrNL8lYsy/4tL/4Lb4IXD4p49Q+YbXtQ60BgmJnHy5sgkcKA6z47U9Mzs3PzpYXy4tLymplbf3SxKlmvMViGevrgBouheItECD5daI5jQLJr4Lbk8K/uPaiFhdwCDhnYj2lAgFo2Alv9I8rd3vHmhpiwjeaZy7EkeQs0zaeRn4ojkN5na+ShT7FN4l086QSF42nR68NO2a9U3bo7BJ4kZEyqaIymX3nyujFLI6ASWpMm7gJdDKqQTDJ87KXGp5Qdkt7vG2pohE3nWx4eY63rdLFYaztU4CH6s+JjEbGDKLAJiMKfPXK8T/vHYK4WEnEypJgSs2WhSmEkOMixpxV2jOQA4soUwL+1fM+tS2CLbsogTy9+RJctmoE7dOzverx2RcRwltoi1UQwQdoGN0hpqohRh6QC/oDb07j86r8+F8jqJTznhmA/2C8/UN+S+rGw=</latexit>

where, fa(x) = ( x2 + x x ≥ 0 Rx2 + x x < 0

<latexit sha1_base64="MQKihgtUtfMQ4JQV24eKnKeSYw=">ACMXicbVBNS+wFE3V50fU0dugkOTxRlaEXQhYLgxqWKo8J0HNLM7UwTWtyqx3K/CU3/hNx40IRt/4JM2MXfh0InJxz703uCVMpDHreozMyOvZnfGJyv37b3pmtjI3f2qSTHOo80Qm+jxkBqRQUEeBEs5TDSwOJZyFl/sD/+watBGJOsFeCs2YdZSIBGdopVblIEDIsbjpgoZ12o9abCVf3Q1C6AhVcDvY9N38YmMtX86Dlx5QeAel/cdzw1AtcuqVqXq1bwh6E/il6RKShy2KvdBO+FZDAq5ZMY0fC/FZsE0Ci6h7waZgZTxS9aBhqWKxWCaxXDjPv1vlTaNEm2PQjpUP3cULDamF4e2MmbYNd+9gfib18gw2m4WQqUZguIfD0WZpJjQXy0LTRwlD1LGNfC/pXyLtOMow3ZtSH431f+SU43ar5X8482q3t+GckWSRLZIX4ZIvskQNySOqEk1vyQJ7Is3PnPDovzutH6YhT9iyQL3De3gFugqkH</latexit><latexit sha1_base64="MQKihgtUtfMQ4JQV24eKnKeSYw=">ACMXicbVBNS+wFE3V50fU0dugkOTxRlaEXQhYLgxqWKo8J0HNLM7UwTWtyqx3K/CU3/hNx40IRt/4JM2MXfh0InJxz703uCVMpDHreozMyOvZnfGJyv37b3pmtjI3f2qSTHOo80Qm+jxkBqRQUEeBEs5TDSwOJZyFl/sD/+watBGJOsFeCs2YdZSIBGdopVblIEDIsbjpgoZ12o9abCVf3Q1C6AhVcDvY9N38YmMtX86Dlx5QeAel/cdzw1AtcuqVqXq1bwh6E/il6RKShy2KvdBO+FZDAq5ZMY0fC/FZsE0Ci6h7waZgZTxS9aBhqWKxWCaxXDjPv1vlTaNEm2PQjpUP3cULDamF4e2MmbYNd+9gfib18gw2m4WQqUZguIfD0WZpJjQXy0LTRwlD1LGNfC/pXyLtOMow3ZtSH431f+SU43ar5X8482q3t+GckWSRLZIX4ZIvskQNySOqEk1vyQJ7Is3PnPDovzutH6YhT9iyQL3De3gFugqkH</latexit><latexit sha1_base64="MQKihgtUtfMQ4JQV24eKnKeSYw=">ACMXicbVBNS+wFE3V50fU0dugkOTxRlaEXQhYLgxqWKo8J0HNLM7UwTWtyqx3K/CU3/hNx40IRt/4JM2MXfh0InJxz703uCVMpDHreozMyOvZnfGJyv37b3pmtjI3f2qSTHOo80Qm+jxkBqRQUEeBEs5TDSwOJZyFl/sD/+watBGJOsFeCs2YdZSIBGdopVblIEDIsbjpgoZ12o9abCVf3Q1C6AhVcDvY9N38YmMtX86Dlx5QeAel/cdzw1AtcuqVqXq1bwh6E/il6RKShy2KvdBO+FZDAq5ZMY0fC/FZsE0Ci6h7waZgZTxS9aBhqWKxWCaxXDjPv1vlTaNEm2PQjpUP3cULDamF4e2MmbYNd+9gfib18gw2m4WQqUZguIfD0WZpJjQXy0LTRwlD1LGNfC/pXyLtOMow3ZtSH431f+SU43ar5X8482q3t+GckWSRLZIX4ZIvskQNySOqEk1vyQJ7Is3PnPDovzutH6YhT9iyQL3De3gFugqkH</latexit><latexit sha1_base64="MQKihgtUtfMQ4JQV24eKnKeSYw=">ACMXicbVBNS+wFE3V50fU0dugkOTxRlaEXQhYLgxqWKo8J0HNLM7UwTWtyqx3K/CU3/hNx40IRt/4JM2MXfh0InJxz703uCVMpDHreozMyOvZnfGJyv37b3pmtjI3f2qSTHOo80Qm+jxkBqRQUEeBEs5TDSwOJZyFl/sD/+watBGJOsFeCs2YdZSIBGdopVblIEDIsbjpgoZ12o9abCVf3Q1C6AhVcDvY9N38YmMtX86Dlx5QeAel/cdzw1AtcuqVqXq1bwh6E/il6RKShy2KvdBO+FZDAq5ZMY0fC/FZsE0Ci6h7waZgZTxS9aBhqWKxWCaxXDjPv1vlTaNEm2PQjpUP3cULDamF4e2MmbYNd+9gfib18gw2m4WQqUZguIfD0WZpJjQXy0LTRwlD1LGNfC/pXyLtOMow3ZtSH431f+SU43ar5X8482q3t+GckWSRLZIX4ZIvskQNySOqEk1vyQJ7Is3PnPDovzutH6YhT9iyQL3De3gFugqkH</latexit>

and, fb(x) = ( x2 − x x ≥ 0 Rx2 − x x < 0

<latexit sha1_base64="SmzCZGqQS5jn9RbDU3jM39gZj0I=">ACL3icbVDLatAFB2leSovp1l2M9QkJAYyQSRQuBQOgyLXUcsBwzGl3Zg0cjZeYqyAj/UTb5lWxCaSjd9i86trXI68DAmXPuvTP3hJkUBj3vlzP3YX5hcWl5xV1dW9/YrG19vDRprjm0eCpTfRUyA1IoaKFACVeZBpaEtrh8Gzit29BG5GqnzjKoJuwvhKx4Ayt1KudBwgFlkxFB3Qc98K9Yv9rEJfqJLbsWbsFtfNw2K3CPpw4wWB+6O6f/HcAFRUVfVqda/hTUHfEr8idVLhold7CKU5wko5JIZ0/G9DLsl0yi4hLEb5AYyxoesDx1LFUvAdMvpvmO6Y5WIxqm2RyGdqs87SpYM0pCW5kwHJjX3kR8z+vkGJ90S6GyHEHx2UNxLimdBIejYQGjnJkCeNa2L9SPmCacbQRuzYE/XKb8ls+F7Df/7Uf3Ur+JYJp/IZ7JHfHJMTsk3ckFahJM78kB+kyfn3nl0/jh/Z6VzTtWzTV7A+fcfjDeoEA=</latexit><latexit sha1_base64="SmzCZGqQS5jn9RbDU3jM39gZj0I=">ACL3icbVDLatAFB2leSovp1l2M9QkJAYyQSRQuBQOgyLXUcsBwzGl3Zg0cjZeYqyAj/UTb5lWxCaSjd9i86trXI68DAmXPuvTP3hJkUBj3vlzP3YX5hcWl5xV1dW9/YrG19vDRprjm0eCpTfRUyA1IoaKFACVeZBpaEtrh8Gzit29BG5GqnzjKoJuwvhKx4Ayt1KudBwgFlkxFB3Qc98K9Yv9rEJfqJLbsWbsFtfNw2K3CPpw4wWB+6O6f/HcAFRUVfVqda/hTUHfEr8idVLhold7CKU5wko5JIZ0/G9DLsl0yi4hLEb5AYyxoesDx1LFUvAdMvpvmO6Y5WIxqm2RyGdqs87SpYM0pCW5kwHJjX3kR8z+vkGJ90S6GyHEHx2UNxLimdBIejYQGjnJkCeNa2L9SPmCacbQRuzYE/XKb8ls+F7Df/7Uf3Ur+JYJp/IZ7JHfHJMTsk3ckFahJM78kB+kyfn3nl0/jh/Z6VzTtWzTV7A+fcfjDeoEA=</latexit><latexit sha1_base64="SmzCZGqQS5jn9RbDU3jM39gZj0I=">ACL3icbVDLatAFB2leSovp1l2M9QkJAYyQSRQuBQOgyLXUcsBwzGl3Zg0cjZeYqyAj/UTb5lWxCaSjd9i86trXI68DAmXPuvTP3hJkUBj3vlzP3YX5hcWl5xV1dW9/YrG19vDRprjm0eCpTfRUyA1IoaKFACVeZBpaEtrh8Gzit29BG5GqnzjKoJuwvhKx4Ayt1KudBwgFlkxFB3Qc98K9Yv9rEJfqJLbsWbsFtfNw2K3CPpw4wWB+6O6f/HcAFRUVfVqda/hTUHfEr8idVLhold7CKU5wko5JIZ0/G9DLsl0yi4hLEb5AYyxoesDx1LFUvAdMvpvmO6Y5WIxqm2RyGdqs87SpYM0pCW5kwHJjX3kR8z+vkGJ90S6GyHEHx2UNxLimdBIejYQGjnJkCeNa2L9SPmCacbQRuzYE/XKb8ls+F7Df/7Uf3Ur+JYJp/IZ7JHfHJMTsk3ckFahJM78kB+kyfn3nl0/jh/Z6VzTtWzTV7A+fcfjDeoEA=</latexit><latexit sha1_base64="SmzCZGqQS5jn9RbDU3jM39gZj0I=">ACL3icbVDLatAFB2leSovp1l2M9QkJAYyQSRQuBQOgyLXUcsBwzGl3Zg0cjZeYqyAj/UTb5lWxCaSjd9i86trXI68DAmXPuvTP3hJkUBj3vlzP3YX5hcWl5xV1dW9/YrG19vDRprjm0eCpTfRUyA1IoaKFACVeZBpaEtrh8Gzit29BG5GqnzjKoJuwvhKx4Ayt1KudBwgFlkxFB3Qc98K9Yv9rEJfqJLbsWbsFtfNw2K3CPpw4wWB+6O6f/HcAFRUVfVqda/hTUHfEr8idVLhold7CKU5wko5JIZ0/G9DLsl0yi4hLEb5AYyxoesDx1LFUvAdMvpvmO6Y5WIxqm2RyGdqs87SpYM0pCW5kwHJjX3kR8z+vkGJ90S6GyHEHx2UNxLimdBIejYQGjnJkCeNa2L9SPmCacbQRuzYE/XKb8ls+F7Df/7Uf3Ur+JYJp/IZ7JHfHJMTsk3ckFahJM78kB+kyfn3nl0/jh/Z6VzTtWzTV7A+fcfjDeoEA=</latexit>

If F has Lipschitz Hessian, Error = O ⇣

n3 T 3 + 1 T 2

⌘ [3]

<latexit sha1_base64="Tn6dgNK7Db1Lz7nIlJu0MYwInY4=">ACPnicbVBNSyNBEO3x2/gV16OXxkRQlDATD3oRhMVFQVDBqJAZQ0+nJtPY0zN01yzEIb9sL/sb9rZHLx4U8erRzsfBr3doXr1XRVe9MJPCoOv+d8bGJyanpmdmS3PzC4tL5eUflybNYcGT2Wqr0NmQAoFDRQo4TrTwJQwlV4+7PvX/0GbUSqLrCbQZCwjhKR4Ayt1Co3jiNa/VWlMTP0RGSGxwLv6BEYI5japodap5pW909CRFu+JFmvFA3O73iwj5bw9rV/Wer0Unxs1qcydolStuzR2AfiXeiFTICGet8j+/nfI8AYVcMmOanpthUDCNgkvolfzcQMb4LetA01LFEjBMTi/R9et0qaRXTRKFdKB+n6iYIkx3S0nQnD2Hz2+uJ3XjPHaC8ohMpyBMWH0W5pJjSfpa0LTRwlF1LGNfC7kp5zGwkaBMv2RC8zyd/JZf1mufWvPN65aA+imOGrJI1skE8sksOyBE5Iw3CyR9yTx7Jk/PXeXCenZdh65gzmlkhH+C8vgF7q1k</latexit><latexit sha1_base64="Tn6dgNK7Db1Lz7nIlJu0MYwInY4=">ACPnicbVBNSyNBEO3x2/gV16OXxkRQlDATD3oRhMVFQVDBqJAZQ0+nJtPY0zN01yzEIb9sL/sb9rZHLx4U8erRzsfBr3doXr1XRVe9MJPCoOv+d8bGJyanpmdmS3PzC4tL5eUflybNYcGT2Wqr0NmQAoFDRQo4TrTwJQwlV4+7PvX/0GbUSqLrCbQZCwjhKR4Ayt1Co3jiNa/VWlMTP0RGSGxwLv6BEYI5japodap5pW909CRFu+JFmvFA3O73iwj5bw9rV/Wer0Unxs1qcydolStuzR2AfiXeiFTICGet8j+/nfI8AYVcMmOanpthUDCNgkvolfzcQMb4LetA01LFEjBMTi/R9et0qaRXTRKFdKB+n6iYIkx3S0nQnD2Hz2+uJ3XjPHaC8ohMpyBMWH0W5pJjSfpa0LTRwlF1LGNfC7kp5zGwkaBMv2RC8zyd/JZf1mufWvPN65aA+imOGrJI1skE8sksOyBE5Iw3CyR9yTx7Jk/PXeXCenZdh65gzmlkhH+C8vgF7q1k</latexit><latexit sha1_base64="Tn6dgNK7Db1Lz7nIlJu0MYwInY4=">ACPnicbVBNSyNBEO3x2/gV16OXxkRQlDATD3oRhMVFQVDBqJAZQ0+nJtPY0zN01yzEIb9sL/sb9rZHLx4U8erRzsfBr3doXr1XRVe9MJPCoOv+d8bGJyanpmdmS3PzC4tL5eUflybNYcGT2Wqr0NmQAoFDRQo4TrTwJQwlV4+7PvX/0GbUSqLrCbQZCwjhKR4Ayt1Co3jiNa/VWlMTP0RGSGxwLv6BEYI5japodap5pW909CRFu+JFmvFA3O73iwj5bw9rV/Wer0Unxs1qcydolStuzR2AfiXeiFTICGet8j+/nfI8AYVcMmOanpthUDCNgkvolfzcQMb4LetA01LFEjBMTi/R9et0qaRXTRKFdKB+n6iYIkx3S0nQnD2Hz2+uJ3XjPHaC8ohMpyBMWH0W5pJjSfpa0LTRwlF1LGNfC7kp5zGwkaBMv2RC8zyd/JZf1mufWvPN65aA+imOGrJI1skE8sksOyBE5Iw3CyR9yTx7Jk/PXeXCenZdh65gzmlkhH+C8vgF7q1k</latexit><latexit sha1_base64="Tn6dgNK7Db1Lz7nIlJu0MYwInY4=">ACPnicbVBNSyNBEO3x2/gV16OXxkRQlDATD3oRhMVFQVDBqJAZQ0+nJtPY0zN01yzEIb9sL/sb9rZHLx4U8erRzsfBr3doXr1XRVe9MJPCoOv+d8bGJyanpmdmS3PzC4tL5eUflybNYcGT2Wqr0NmQAoFDRQo4TrTwJQwlV4+7PvX/0GbUSqLrCbQZCwjhKR4Ayt1Co3jiNa/VWlMTP0RGSGxwLv6BEYI5japodap5pW909CRFu+JFmvFA3O73iwj5bw9rV/Wer0Unxs1qcydolStuzR2AfiXeiFTICGet8j+/nfI8AYVcMmOanpthUDCNgkvolfzcQMb4LetA01LFEjBMTi/R9et0qaRXTRKFdKB+n6iYIkx3S0nQnD2Hz2+uJ3XjPHaC8ohMpyBMWH0W5pJjSfpa0LTRwlF1LGNfC7kp5zGwkaBMv2RC8zyd/JZf1mufWvPN65aA+imOGrJI1skE8sksOyBE5Iw3CyR9yTx7Jk/PXeXCenZdh65gzmlkhH+C8vgF7q1k</latexit>
slide-13
SLIDE 13

rfa(x) = ( x 0 x < 0

<latexit sha1_base64="Od1Qas2bnXEvIblYUXsgfzitCUY=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARh3Yuds98gPscpkz+6K+L/m9Xgbh6N6+87vYc31/jD10Sz7KsLB0yhW36g6L/gVeASqkqHqn/OSHCctilIYJ0Lrlualp56AMZwLtCJnGFNgNdLFloYQYdTsf7n1PtywT0ihR9khDh+xPRw6x1v04sJ0xmGs9rg3I/7RWZqKDds5lmhmUbPRlAlqEjoIkYZcITOibwEwxe2slF2DAmZs1CUbgje+8l/QrFU9t+qd71WOa0Uc2SDbJId4pF9ckxOSZ0CMP5IW8kXfn0Xl1PpzPUeuEU3jWya9yvr4B1Gysdg=</latexit><latexit sha1_base64="Od1Qas2bnXEvIblYUXsgfzitCUY=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARh3Yuds98gPscpkz+6K+L/m9Xgbh6N6+87vYc31/jD10Sz7KsLB0yhW36g6L/gVeASqkqHqn/OSHCctilIYJ0Lrlualp56AMZwLtCJnGFNgNdLFloYQYdTsf7n1PtywT0ihR9khDh+xPRw6x1v04sJ0xmGs9rg3I/7RWZqKDds5lmhmUbPRlAlqEjoIkYZcITOibwEwxe2slF2DAmZs1CUbgje+8l/QrFU9t+qd71WOa0Uc2SDbJId4pF9ckxOSZ0CMP5IW8kXfn0Xl1PpzPUeuEU3jWya9yvr4B1Gysdg=</latexit><latexit sha1_base64="Od1Qas2bnXEvIblYUXsgfzitCUY=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARh3Yuds98gPscpkz+6K+L/m9Xgbh6N6+87vYc31/jD10Sz7KsLB0yhW36g6L/gVeASqkqHqn/OSHCctilIYJ0Lrlualp56AMZwLtCJnGFNgNdLFloYQYdTsf7n1PtywT0ihR9khDh+xPRw6x1v04sJ0xmGs9rg3I/7RWZqKDds5lmhmUbPRlAlqEjoIkYZcITOibwEwxe2slF2DAmZs1CUbgje+8l/QrFU9t+qd71WOa0Uc2SDbJId4pF9ckxOSZ0CMP5IW8kXfn0Xl1PpzPUeuEU3jWya9yvr4B1Gysdg=</latexit><latexit sha1_base64="Od1Qas2bnXEvIblYUXsgfzitCUY=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARh3Yuds98gPscpkz+6K+L/m9Xgbh6N6+87vYc31/jD10Sz7KsLB0yhW36g6L/gVeASqkqHqn/OSHCctilIYJ0Lrlualp56AMZwLtCJnGFNgNdLFloYQYdTsf7n1PtywT0ihR9khDh+xPRw6x1v04sJ0xmGs9rg3I/7RWZqKDds5lmhmUbPRlAlqEjoIkYZcITOibwEwxe2slF2DAmZs1CUbgje+8l/QrFU9t+qd71WOa0Uc2SDbJId4pF9ckxOSZ0CMP5IW8kXfn0Xl1PpzPUeuEU3jWya9yvr4B1Gysdg=</latexit>

Proof sketch

  • Consider the function gradients
  • When x is small, the gradient is dominated by the gradients of linear terms
  • These are Rademacher variables (but not independent)
  • 2x + 1

2Rx + 1 2x − 1 2Rx − 1 rfb(x) = ( x 0 x < 0

<latexit sha1_base64="4i2tqHSPLVcd2qDNVRyeEp41e5Q=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARp1g5273yA+wy2XO7Iv6vuT3ehmEo3v7zu9iz/X9MfbQLfkow8LSKVfcqjs+hd4BaiQouqd8pMfJiyLURomQOuW56amnYMynAm0I2QaU2A30MWhRJi1O18uPc93bJMSKNE2SMNHbI/HTnEWvfjwHbGYK71uDYg/9NamYkO2jmXaWZQstFHUSaoSegRBpyhcyIvgXAFLezUnYNCpixUZdsCN74yn9Bs1b13Kp3vlc5rhVxzJENskl2iEf2yTE5JXSIw8kBfyRt6dR+fV+XA+R60TuFZJ7/K+foG1jSsdw=</latexit><latexit sha1_base64="4i2tqHSPLVcd2qDNVRyeEp41e5Q=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARp1g5273yA+wy2XO7Iv6vuT3ehmEo3v7zu9iz/X9MfbQLfkow8LSKVfcqjs+hd4BaiQouqd8pMfJiyLURomQOuW56amnYMynAm0I2QaU2A30MWhRJi1O18uPc93bJMSKNE2SMNHbI/HTnEWvfjwHbGYK71uDYg/9NamYkO2jmXaWZQstFHUSaoSegRBpyhcyIvgXAFLezUnYNCpixUZdsCN74yn9Bs1b13Kp3vlc5rhVxzJENskl2iEf2yTE5JXSIw8kBfyRt6dR+fV+XA+R60TuFZJ7/K+foG1jSsdw=</latexit><latexit sha1_base64="4i2tqHSPLVcd2qDNVRyeEp41e5Q=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARp1g5273yA+wy2XO7Iv6vuT3ehmEo3v7zu9iz/X9MfbQLfkow8LSKVfcqjs+hd4BaiQouqd8pMfJiyLURomQOuW56amnYMynAm0I2QaU2A30MWhRJi1O18uPc93bJMSKNE2SMNHbI/HTnEWvfjwHbGYK71uDYg/9NamYkO2jmXaWZQstFHUSaoSegRBpyhcyIvgXAFLezUnYNCpixUZdsCN74yn9Bs1b13Kp3vlc5rhVxzJENskl2iEf2yTE5JXSIw8kBfyRt6dR+fV+XA+R60TuFZJ7/K+foG1jSsdw=</latexit><latexit sha1_base64="4i2tqHSPLVcd2qDNVRyeEp41e5Q=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARp1g5273yA+wy2XO7Iv6vuT3ehmEo3v7zu9iz/X9MfbQLfkow8LSKVfcqjs+hd4BaiQouqd8pMfJiyLURomQOuW56amnYMynAm0I2QaU2A30MWhRJi1O18uPc93bJMSKNE2SMNHbI/HTnEWvfjwHbGYK71uDYg/9NamYkO2jmXaWZQstFHUSaoSegRBpyhcyIvgXAFLezUnYNCpixUZdsCN74yn9Bs1b13Kp3vlc5rhVxzJENskl2iEf2yTE5JXSIw8kBfyRt6dR+fV+XA+R60TuFZJ7/K+foG1jSsdw=</latexit>

For i ≤ n

4 , |xi| ≥ Cα

√ i

<latexit sha1_base64="3u8MRuhVHK9bmjS8LAJYZ0eX51w=">ACH3icbVDLSsNAFJ3UV62vqks3g63gQkpSRAtuCgVxWcE+oAlhMp20QyeTdGYiljR/4sZfceNCEXHXv3H6WGjrWR3OuZd7vEiRqUyzYmRWVvf2NzKbud2dvf2D/KHR0ZxgKTBg5ZKNoekoRThqKkbakSAo8BhpeYPa1G89EiFpyB/UKCJOgHqc+hQjpSU3f3UbClikNiNDaPsC4YSnyWV6Ae0bOH5y6djuadmIxb1EbTlUKiEpkU3XzBL5gxwlVgLUgAL1N38t90NcRwQrjBDUnYsM1JOgoSimJE0Z8eSRAgPUI90NOUoINJZv+l8EwrXejrpH7IFZypvzcSFEg5Cjw9GSDVl8veVPzP68TKrzgJ5VGsCMfzQ37MoArhtCzYpYJgxUaICyozgpxH+mWlK40p0uwl9eJc1yTJL1n25UK0s6siCE3AKzoEFrkEV3IE6aAMnsEreAcfxovxZnwaX/PRjLHYOQZ/YEx+AODLojs=</latexit><latexit sha1_base64="3u8MRuhVHK9bmjS8LAJYZ0eX51w=">ACH3icbVDLSsNAFJ3UV62vqks3g63gQkpSRAtuCgVxWcE+oAlhMp20QyeTdGYiljR/4sZfceNCEXHXv3H6WGjrWR3OuZd7vEiRqUyzYmRWVvf2NzKbud2dvf2D/KHR0ZxgKTBg5ZKNoekoRThqKkbakSAo8BhpeYPa1G89EiFpyB/UKCJOgHqc+hQjpSU3f3UbClikNiNDaPsC4YSnyWV6Ae0bOH5y6djuadmIxb1EbTlUKiEpkU3XzBL5gxwlVgLUgAL1N38t90NcRwQrjBDUnYsM1JOgoSimJE0Z8eSRAgPUI90NOUoINJZv+l8EwrXejrpH7IFZypvzcSFEg5Cjw9GSDVl8veVPzP68TKrzgJ5VGsCMfzQ37MoArhtCzYpYJgxUaICyozgpxH+mWlK40p0uwl9eJc1yTJL1n25UK0s6siCE3AKzoEFrkEV3IE6aAMnsEreAcfxovxZnwaX/PRjLHYOQZ/YEx+AODLojs=</latexit><latexit sha1_base64="3u8MRuhVHK9bmjS8LAJYZ0eX51w=">ACH3icbVDLSsNAFJ3UV62vqks3g63gQkpSRAtuCgVxWcE+oAlhMp20QyeTdGYiljR/4sZfceNCEXHXv3H6WGjrWR3OuZd7vEiRqUyzYmRWVvf2NzKbud2dvf2D/KHR0ZxgKTBg5ZKNoekoRThqKkbakSAo8BhpeYPa1G89EiFpyB/UKCJOgHqc+hQjpSU3f3UbClikNiNDaPsC4YSnyWV6Ae0bOH5y6djuadmIxb1EbTlUKiEpkU3XzBL5gxwlVgLUgAL1N38t90NcRwQrjBDUnYsM1JOgoSimJE0Z8eSRAgPUI90NOUoINJZv+l8EwrXejrpH7IFZypvzcSFEg5Cjw9GSDVl8veVPzP68TKrzgJ5VGsCMfzQ37MoArhtCzYpYJgxUaICyozgpxH+mWlK40p0uwl9eJc1yTJL1n25UK0s6siCE3AKzoEFrkEV3IE6aAMnsEreAcfxovxZnwaX/PRjLHYOQZ/YEx+AODLojs=</latexit><latexit sha1_base64="3u8MRuhVHK9bmjS8LAJYZ0eX51w=">ACH3icbVDLSsNAFJ3UV62vqks3g63gQkpSRAtuCgVxWcE+oAlhMp20QyeTdGYiljR/4sZfceNCEXHXv3H6WGjrWR3OuZd7vEiRqUyzYmRWVvf2NzKbud2dvf2D/KHR0ZxgKTBg5ZKNoekoRThqKkbakSAo8BhpeYPa1G89EiFpyB/UKCJOgHqc+hQjpSU3f3UbClikNiNDaPsC4YSnyWV6Ae0bOH5y6djuadmIxb1EbTlUKiEpkU3XzBL5gxwlVgLUgAL1N38t90NcRwQrjBDUnYsM1JOgoSimJE0Z8eSRAgPUI90NOUoINJZv+l8EwrXejrpH7IFZypvzcSFEg5Cjw9GSDVl8veVPzP68TKrzgJ5VGsCMfzQ37MoArhtCzYpYJgxUaICyozgpxH+mWlK40p0uwl9eJc1yTJL1n25UK0s6siCE3AKzoEFrkEV3IE6aAMnsEreAcfxovxZnwaX/PRjLHYOQZ/YEx+AODLojs=</latexit>
slide-14
SLIDE 14

Proof sketch

  • The sum of gradients from linear terms = 0
  • The sum of gradients from quadratic terms

rfa(x) = ( x 0 x < 0

<latexit sha1_base64="Od1Qas2bnXEvIblYUXsgfzitCUY=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARh3Yuds98gPscpkz+6K+L/m9Xgbh6N6+87vYc31/jD10Sz7KsLB0yhW36g6L/gVeASqkqHqn/OSHCctilIYJ0Lrlualp56AMZwLtCJnGFNgNdLFloYQYdTsf7n1PtywT0ihR9khDh+xPRw6x1v04sJ0xmGs9rg3I/7RWZqKDds5lmhmUbPRlAlqEjoIkYZcITOibwEwxe2slF2DAmZs1CUbgje+8l/QrFU9t+qd71WOa0Uc2SDbJId4pF9ckxOSZ0CMP5IW8kXfn0Xl1PpzPUeuEU3jWya9yvr4B1Gysdg=</latexit><latexit sha1_base64="Od1Qas2bnXEvIblYUXsgfzitCUY=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARh3Yuds98gPscpkz+6K+L/m9Xgbh6N6+87vYc31/jD10Sz7KsLB0yhW36g6L/gVeASqkqHqn/OSHCctilIYJ0Lrlualp56AMZwLtCJnGFNgNdLFloYQYdTsf7n1PtywT0ihR9khDh+xPRw6x1v04sJ0xmGs9rg3I/7RWZqKDds5lmhmUbPRlAlqEjoIkYZcITOibwEwxe2slF2DAmZs1CUbgje+8l/QrFU9t+qd71WOa0Uc2SDbJId4pF9ckxOSZ0CMP5IW8kXfn0Xl1PpzPUeuEU3jWya9yvr4B1Gysdg=</latexit><latexit sha1_base64="Od1Qas2bnXEvIblYUXsgfzitCUY=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARh3Yuds98gPscpkz+6K+L/m9Xgbh6N6+87vYc31/jD10Sz7KsLB0yhW36g6L/gVeASqkqHqn/OSHCctilIYJ0Lrlualp56AMZwLtCJnGFNgNdLFloYQYdTsf7n1PtywT0ihR9khDh+xPRw6x1v04sJ0xmGs9rg3I/7RWZqKDds5lmhmUbPRlAlqEjoIkYZcITOibwEwxe2slF2DAmZs1CUbgje+8l/QrFU9t+qd71WOa0Uc2SDbJId4pF9ckxOSZ0CMP5IW8kXfn0Xl1PpzPUeuEU3jWya9yvr4B1Gysdg=</latexit><latexit sha1_base64="Od1Qas2bnXEvIblYUXsgfzitCUY=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARh3Yuds98gPscpkz+6K+L/m9Xgbh6N6+87vYc31/jD10Sz7KsLB0yhW36g6L/gVeASqkqHqn/OSHCctilIYJ0Lrlualp56AMZwLtCJnGFNgNdLFloYQYdTsf7n1PtywT0ihR9khDh+xPRw6x1v04sJ0xmGs9rg3I/7RWZqKDds5lmhmUbPRlAlqEjoIkYZcITOibwEwxe2slF2DAmZs1CUbgje+8l/QrFU9t+qd71WOa0Uc2SDbJId4pF9ckxOSZ0CMP5IW8kXfn0Xl1PpzPUeuEU3jWya9yvr4B1Gysdg=</latexit>

2x + 1 2Rx + 1 2x − 1 2Rx − 1 rfb(x) = ( x 0 x < 0

<latexit sha1_base64="4i2tqHSPLVcd2qDNVRyeEp41e5Q=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARp1g5273yA+wy2XO7Iv6vuT3ehmEo3v7zu9iz/X9MfbQLfkow8LSKVfcqjs+hd4BaiQouqd8pMfJiyLURomQOuW56amnYMynAm0I2QaU2A30MWhRJi1O18uPc93bJMSKNE2SMNHbI/HTnEWvfjwHbGYK71uDYg/9NamYkO2jmXaWZQstFHUSaoSegRBpyhcyIvgXAFLezUnYNCpixUZdsCN74yn9Bs1b13Kp3vlc5rhVxzJENskl2iEf2yTE5JXSIw8kBfyRt6dR+fV+XA+R60TuFZJ7/K+foG1jSsdw=</latexit><latexit sha1_base64="4i2tqHSPLVcd2qDNVRyeEp41e5Q=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARp1g5273yA+wy2XO7Iv6vuT3ehmEo3v7zu9iz/X9MfbQLfkow8LSKVfcqjs+hd4BaiQouqd8pMfJiyLURomQOuW56amnYMynAm0I2QaU2A30MWhRJi1O18uPc93bJMSKNE2SMNHbI/HTnEWvfjwHbGYK71uDYg/9NamYkO2jmXaWZQstFHUSaoSegRBpyhcyIvgXAFLezUnYNCpixUZdsCN74yn9Bs1b13Kp3vlc5rhVxzJENskl2iEf2yTE5JXSIw8kBfyRt6dR+fV+XA+R60TuFZJ7/K+foG1jSsdw=</latexit><latexit sha1_base64="4i2tqHSPLVcd2qDNVRyeEp41e5Q=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARp1g5273yA+wy2XO7Iv6vuT3ehmEo3v7zu9iz/X9MfbQLfkow8LSKVfcqjs+hd4BaiQouqd8pMfJiyLURomQOuW56amnYMynAm0I2QaU2A30MWhRJi1O18uPc93bJMSKNE2SMNHbI/HTnEWvfjwHbGYK71uDYg/9NamYkO2jmXaWZQstFHUSaoSegRBpyhcyIvgXAFLezUnYNCpixUZdsCN74yn9Bs1b13Kp3vlc5rhVxzJENskl2iEf2yTE5JXSIw8kBfyRt6dR+fV+XA+R60TuFZJ7/K+foG1jSsdw=</latexit><latexit sha1_base64="4i2tqHSPLVcd2qDNVRyeEp41e5Q=">ACN3icbVBNSwMxEM36bf2qevQSLIpeym4R9KAgePEkFWwVuqXMZmdrMJvdJlmxLP4rL/4Nb3rxoIhX/4FpuwetDiQ83puXzLwgFVwb1312Jianpmdm5+ZLC4tLyvl1bWmTjLFsMESkairADQKLrFhuBF4lSqEOB4GdycDPTLW1SaJ/LC9FNsx9CVPOIMjKU65TNfQiCARp1g5273yA+wy2XO7Iv6vuT3ehmEo3v7zu9iz/X9MfbQLfkow8LSKVfcqjs+hd4BaiQouqd8pMfJiyLURomQOuW56amnYMynAm0I2QaU2A30MWhRJi1O18uPc93bJMSKNE2SMNHbI/HTnEWvfjwHbGYK71uDYg/9NamYkO2jmXaWZQstFHUSaoSegRBpyhcyIvgXAFLezUnYNCpixUZdsCN74yn9Bs1b13Kp3vlc5rhVxzJENskl2iEf2yTE5JXSIw8kBfyRt6dR+fV+XA+R60TuFZJ7/K+foG1jSsdw=</latexit>

xn x1 = α Pn

i=1 rfσ(i)(xi)

<latexit sha1_base64="pqRsmjQ1bCbIVEsC7Ogaj8T7ksM=">ACH3icbVA9TxtBEJ0j4cvhwyQlzYKFZAqsW4qEBslSmlQIpNg+cxpbr1nr9jdO+3uIVsnN/yOFPBX0lAEIZTO/4a1TREgTxrp6b0ZzcxLcimsC8NJsPDh4+LS8spq5dPa+sZmdetz2aFYbzFMpmZiwQtl0LzlhNO8ovcFSJ5OfJ1fepf37NjRWZ/ulGOe8q7GuRCobOS3H16zDWB8OYHkco8wFGtlBxKY7p+FJHGhOJI3LyIq+wrYHxNSH8aC7MfVWtgIZyDvCX0htebOyc0tAJzG1b9RL2OF4toxidZ2aJi7bonGCSb5uBIVlufIrDPO5qVNx2y9l/Y7LnlR5JM+NLOzJT/50oUVk7UonvVOgG9q03Ff/ndQqXHnVLofPCc3mi9JCEpeRaVikJwxnTo48QWaEv5WwARpkzkda8SHQty+/J+3DBg0b9IzWmkcwxwpswy7UgcI3aMIPOIUWMPgFv+EPAR3wX3wGDzNWxeCl5kv8ArB5BnHtqOe</latexit><latexit sha1_base64="wC3WZWkYM5O0msGFAgm+wdJIWD8=">ACH3icbVC7ahtBFJ1/JDlR2SnTDO2CMiFxY6LWI1A4MZVUMB6gFZ7o5mpcEzs8vMrJFY1OQbUgU3/hU3KRKCcae/8ehR+HXgwuGce7n3nigV3Fjfn3lrH9Y3NrcK28Wd3b39j6WDw7ZJMk1ZiyYi0d0IDBNcsZblVrBuqhnISLBOdH0x9zs3TBueqCs7SVlfwlDxmFOwTgpLX8ehOh2HpB6ASEcQmEyGOa+T6Q8VKIgE4DjMA8OHEir8ZIpxZRxyfBKWyn7VXwC/JWRFyo2jbz9/W53m2HpMRgkNJNMWSrAmB7xU9vPQVtOBZsWg8ywFOg1DFnPUQWSmX6+G+KvzhlgONEu1IWL9TnEzlIYyYycp0S7Mi89ubie14vs3Gtn3OVZpYpulwUZwLbBM/DwgOuGbVi4ghQzd2tmI5A7Uu0qILgbx+S1pn1WJXyXfSblRQ0sU0Gd0jCqIoHPUQJeoiVqIolt0j/6if96d98f7z0sW9e81cwn9ALe7AluJqTa</latexit><latexit sha1_base64="wC3WZWkYM5O0msGFAgm+wdJIWD8=">ACH3icbVC7ahtBFJ1/JDlR2SnTDO2CMiFxY6LWI1A4MZVUMB6gFZ7o5mpcEzs8vMrJFY1OQbUgU3/hU3KRKCcae/8ehR+HXgwuGce7n3nigV3Fjfn3lrH9Y3NrcK28Wd3b39j6WDw7ZJMk1ZiyYi0d0IDBNcsZblVrBuqhnISLBOdH0x9zs3TBueqCs7SVlfwlDxmFOwTgpLX8ehOh2HpB6ASEcQmEyGOa+T6Q8VKIgE4DjMA8OHEir8ZIpxZRxyfBKWyn7VXwC/JWRFyo2jbz9/W53m2HpMRgkNJNMWSrAmB7xU9vPQVtOBZsWg8ywFOg1DFnPUQWSmX6+G+KvzhlgONEu1IWL9TnEzlIYyYycp0S7Mi89ubie14vs3Gtn3OVZpYpulwUZwLbBM/DwgOuGbVi4ghQzd2tmI5A7Uu0qILgbx+S1pn1WJXyXfSblRQ0sU0Gd0jCqIoHPUQJeoiVqIolt0j/6if96d98f7z0sW9e81cwn9ALe7AluJqTa</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="aNtiJdP4klTa9Jb9ZsW2Gb+iW0=">ACFHicbZBLSwMxFIXv+Kz1Vd26CRahXVhmXKibguDGZQWrQqcOd9JMG0wyQ5KRlqH/xI1/xY0LRcSd/8a0duHrQODjnISbe+JMcGN9/8Obm19YXFourZRX19Y3Nitba5cmzTVlbZqKVF/HaJjgirUt4JdZ5qhjAW7im9PJ/nVHdOGp+rCjLWldhXPOEUrbOiyuEwUvDKGiGKLIBhiaXUcGbwfhGhQpjgSJitDwvsQar48JqQ0jTupRpeo3/KnIXwhmUIWZWlHlPeylNJdMWSrQmE7gZ7ZboLacCjYuh7lhGdJb7LOQ4WSmW4x3W9M9pzTI0mq3VGWTN3vLwqUxoxk7G5KtAPzO5uY/2Wd3CbH3YKrLdM0a9BS6ITcmkLNLjmlErRg6Qau7+SugANVLrKi27EoLfK/+Fy4NG4DeCcx9KsAO7UIMAjuAEzqAFbaBwD4/wDC/eg/fkvX7VNefNetuGH/LePgFn1KBQ</latexit><latexit sha1_base64="aNtiJdP4klTa9Jb9ZsW2Gb+iW0=">ACFHicbZBLSwMxFIXv+Kz1Vd26CRahXVhmXKibguDGZQWrQqcOd9JMG0wyQ5KRlqH/xI1/xY0LRcSd/8a0duHrQODjnISbe+JMcGN9/8Obm19YXFourZRX19Y3Nitba5cmzTVlbZqKVF/HaJjgirUt4JdZ5qhjAW7im9PJ/nVHdOGp+rCjLWldhXPOEUrbOiyuEwUvDKGiGKLIBhiaXUcGbwfhGhQpjgSJitDwvsQar48JqQ0jTupRpeo3/KnIXwhmUIWZWlHlPeylNJdMWSrQmE7gZ7ZboLacCjYuh7lhGdJb7LOQ4WSmW4x3W9M9pzTI0mq3VGWTN3vLwqUxoxk7G5KtAPzO5uY/2Wd3CbH3YKrLdM0a9BS6ITcmkLNLjmlErRg6Qau7+SugANVLrKi27EoLfK/+Fy4NG4DeCcx9KsAO7UIMAjuAEzqAFbaBwD4/wDC/eg/fkvX7VNefNetuGH/LePgFn1KBQ</latexit><latexit sha1_base64="ibe79HSNFfauURKhBAY3h3lbczg=">ACH3icbVBNS8NAEN3Ur1q/qh69LBahHiyJB+1FELx4rGC10MQw2W7apbubsLuRlpB/4sW/4sWDIuLNf+O29qDWBwOP92aYmRelnGnjup9OaWFxaXmlvFpZW9/Y3Kpu79zoJFOEtknCE9WJQFPOJG0bZjtpIqCiDi9jYXE/2nirNEnltxikNBPQlixkBY6WwejIK5dEo9M584OkAfJ2JMGdnXnEnfQkRBxyHua9ZX0CdHRY10chw4dhteY23CnwPFmpIZmaIXVD7+XkExQaQgHrbuem5ogB2UY4bSo+JmKZAh9GnXUgmC6iCf/lfgA6v0cJwoW9LgqfpzIgeh9VhEtlOAGei/3kT8z+tmJm4GOZNpZqgk34vijGOT4ElYuMcUJYaPLQGimL0VkwEoIMZGWrEheH9fnic3xw3PbXhXbu28OYujPbQPqojD52ic3SJWqiNCHpAT+gFvTqPzrPz5rx/t5ac2cwu+gXn8wtT76HM</latexit><latexit sha1_base64="AKUykt1XTaNONDgDhxs25aVnhUw=">ACH3icbVDLSgMxFM34rPVdekmWIS6sExcaDeC4MZlBdsKnTrcSTNtMkMSUZahv6JG3/FjQtFxF3/xvSx8HXgwuGce7n3nigV3FjfH3sLi0vLK6uFteL6xubWdmlnt2mSTFPWoIlI9G0EhgmuWMNyK9htqhnISLBWdH858VsPTBueqBs7TFlHQk/xmFOwTgpLp4NQHQ9Cch6ASPsQmEyGOT8nozsVKIgE4DjMA8N7Eir8aIRxZRByfBSWyn7VnwL/JWROymiOelj6DLoJzSRTlgowpk381HZy0JZTwUbFIDMsBXoPdZ2VIFkpNP/xvhQ6d0cZxoV8riqfp9IgdpzFBGrlOC7Zvf3kT8z2tnNq51cq7SzDJFZ4viTGCb4ElYuMs1o1YMHQGqubsV0z5oNZFWnQhkN8v/yXNkyrxq+SalC9q8zgKaB8doAoi6AxdoCtURw1E0SN6Rq/ozXvyXrx372PWuDNZ/bQD3jL1Q/oc0=</latexit><latexit sha1_base64="wC3WZWkYM5O0msGFAgm+wdJIWD8=">ACH3icbVC7ahtBFJ1/JDlR2SnTDO2CMiFxY6LWI1A4MZVUMB6gFZ7o5mpcEzs8vMrJFY1OQbUgU3/hU3KRKCcae/8ehR+HXgwuGce7n3nigV3Fjfn3lrH9Y3NrcK28Wd3b39j6WDw7ZJMk1ZiyYi0d0IDBNcsZblVrBuqhnISLBOdH0x9zs3TBueqCs7SVlfwlDxmFOwTgpLX8ehOh2HpB6ASEcQmEyGOa+T6Q8VKIgE4DjMA8OHEir8ZIpxZRxyfBKWyn7VXwC/JWRFyo2jbz9/W53m2HpMRgkNJNMWSrAmB7xU9vPQVtOBZsWg8ywFOg1DFnPUQWSmX6+G+KvzhlgONEu1IWL9TnEzlIYyYycp0S7Mi89ubie14vs3Gtn3OVZpYpulwUZwLbBM/DwgOuGbVi4ghQzd2tmI5A7Uu0qILgbx+S1pn1WJXyXfSblRQ0sU0Gd0jCqIoHPUQJeoiVqIolt0j/6if96d98f7z0sW9e81cwn9ALe7AluJqTa</latexit><latexit sha1_base64="wC3WZWkYM5O0msGFAgm+wdJIWD8=">ACH3icbVC7ahtBFJ1/JDlR2SnTDO2CMiFxY6LWI1A4MZVUMB6gFZ7o5mpcEzs8vMrJFY1OQbUgU3/hU3KRKCcae/8ehR+HXgwuGce7n3nigV3Fjfn3lrH9Y3NrcK28Wd3b39j6WDw7ZJMk1ZiyYi0d0IDBNcsZblVrBuqhnISLBOdH0x9zs3TBueqCs7SVlfwlDxmFOwTgpLX8ehOh2HpB6ASEcQmEyGOa+T6Q8VKIgE4DjMA8OHEir8ZIpxZRxyfBKWyn7VXwC/JWRFyo2jbz9/W53m2HpMRgkNJNMWSrAmB7xU9vPQVtOBZsWg8ywFOg1DFnPUQWSmX6+G+KvzhlgONEu1IWL9TnEzlIYyYycp0S7Mi89ubie14vs3Gtn3OVZpYpulwUZwLbBM/DwgOuGbVi4ghQzd2tmI5A7Uu0qILgbx+S1pn1WJXyXfSblRQ0sU0Gd0jCqIoHPUQJeoiVqIolt0j/6if96d98f7z0sW9e81cwn9ALe7AluJqTa</latexit><latexit sha1_base64="wC3WZWkYM5O0msGFAgm+wdJIWD8=">ACH3icbVC7ahtBFJ1/JDlR2SnTDO2CMiFxY6LWI1A4MZVUMB6gFZ7o5mpcEzs8vMrJFY1OQbUgU3/hU3KRKCcae/8ehR+HXgwuGce7n3nigV3Fjfn3lrH9Y3NrcK28Wd3b39j6WDw7ZJMk1ZiyYi0d0IDBNcsZblVrBuqhnISLBOdH0x9zs3TBueqCs7SVlfwlDxmFOwTgpLX8ehOh2HpB6ASEcQmEyGOa+T6Q8VKIgE4DjMA8OHEir8ZIpxZRxyfBKWyn7VXwC/JWRFyo2jbz9/W53m2HpMRgkNJNMWSrAmB7xU9vPQVtOBZsWg8ywFOg1DFnPUQWSmX6+G+KvzhlgONEu1IWL9TnEzlIYyYycp0S7Mi89ubie14vs3Gtn3OVZpYpulwUZwLbBM/DwgOuGbVi4ghQzd2tmI5A7Uu0qILgbx+S1pn1WJXyXfSblRQ0sU0Gd0jCqIoHPUQJeoiVqIolt0j/6if96d98f7z0sW9e81cwn9ALe7AluJqTa</latexit><latexit sha1_base64="wC3WZWkYM5O0msGFAgm+wdJIWD8=">ACH3icbVC7ahtBFJ1/JDlR2SnTDO2CMiFxY6LWI1A4MZVUMB6gFZ7o5mpcEzs8vMrJFY1OQbUgU3/hU3KRKCcae/8ehR+HXgwuGce7n3nigV3Fjfn3lrH9Y3NrcK28Wd3b39j6WDw7ZJMk1ZiyYi0d0IDBNcsZblVrBuqhnISLBOdH0x9zs3TBueqCs7SVlfwlDxmFOwTgpLX8ehOh2HpB6ASEcQmEyGOa+T6Q8VKIgE4DjMA8OHEir8ZIpxZRxyfBKWyn7VXwC/JWRFyo2jbz9/W53m2HpMRgkNJNMWSrAmB7xU9vPQVtOBZsWg8ywFOg1DFnPUQWSmX6+G+KvzhlgONEu1IWL9TnEzlIYyYycp0S7Mi89ubie14vs3Gtn3OVZpYpulwUZwLbBM/DwgOuGbVi4ghQzd2tmI5A7Uu0qILgbx+S1pn1WJXyXfSblRQ0sU0Gd0jCqIoHPUQJeoiVqIolt0j/6if96d98f7z0sW9e81cwn9ALe7AluJqTa</latexit><latexit sha1_base64="AKUykt1XTaNONDgDhxs25aVnhUw=">ACH3icbVDLSgMxFM34rPVdekmWIS6sExcaDeC4MZlBdsKnTrcSTNtMkMSUZahv6JG3/FjQtFxF3/xvSx8HXgwuGce7n3nigV3FjfH3sLi0vLK6uFteL6xubWdmlnt2mSTFPWoIlI9G0EhgmuWMNyK9htqhnISLBWdH858VsPTBueqBs7TFlHQk/xmFOwTgpLp4NQHQ9Cch6ASPsQmEyGOT8nozsVKIgE4DjMA8N7Eir8aIRxZRByfBSWyn7VnwL/JWROymiOelj6DLoJzSRTlgowpk381HZy0JZTwUbFIDMsBXoPdZ2VIFkpNP/xvhQ6d0cZxoV8riqfp9IgdpzFBGrlOC7Zvf3kT8z2tnNq51cq7SzDJFZ4viTGCb4ElYuMs1o1YMHQGqubsV0z5oNZFWnQhkN8v/yXNkyrxq+SalC9q8zgKaB8doAoi6AxdoCtURw1E0SN6Rq/ozXvyXrx372PWuDNZ/bQD3jL1Q/oc0=</latexit>

X

xi<0

<latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit><latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit><latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="VmSdpJnIqoPNIfkVFuEH2jJZP4=">AB6HicbZBLSwMxFIXv1FetVatbN8EiuCoZN7pwIbhxWcG2QmcYMmDU0yQx5iGfo3LhQxF/kzn9j+lho64HAxzkJ9+akheDGYvwdVDY2t7Z3qru1vfr+wWHjqN41udOUdWgucv2YEsMEV6xjuRXsdCMyFSwXjq+neW9J6YNz9WDnRQslmSoeMYpsd6KIuNkUj4n/BpPk0YTt/BcaB3CJTRhqXbS+IoGOXWSKUsFMaYf4sLGJdGWU8GmtcgZVhA6JkPW96iIZCYu5ztP0Zl3BijLtT/Korn7+0VJpDETmfqbktiRWc1m5n9Z39nsKi65Kpxli4GZU4gm6NZAWjANaNWTDwQqrnfFdER0YRaX1PNlxCufnkduhetELfCewxVOIFTOIcQLuEG7qANHaBQwAu8wXvgtfgY1FXJVj2dgx/FHz+AL0KkEg=</latexit><latexit sha1_base64="VmSdpJnIqoPNIfkVFuEH2jJZP4=">AB6HicbZBLSwMxFIXv1FetVatbN8EiuCoZN7pwIbhxWcG2QmcYMmDU0yQx5iGfo3LhQxF/kzn9j+lho64HAxzkJ9+akheDGYvwdVDY2t7Z3qru1vfr+wWHjqN41udOUdWgucv2YEsMEV6xjuRXsdCMyFSwXjq+neW9J6YNz9WDnRQslmSoeMYpsd6KIuNkUj4n/BpPk0YTt/BcaB3CJTRhqXbS+IoGOXWSKUsFMaYf4sLGJdGWU8GmtcgZVhA6JkPW96iIZCYu5ztP0Zl3BijLtT/Korn7+0VJpDETmfqbktiRWc1m5n9Z39nsKi65Kpxli4GZU4gm6NZAWjANaNWTDwQqrnfFdER0YRaX1PNlxCufnkduhetELfCewxVOIFTOIcQLuEG7qANHaBQwAu8wXvgtfgY1FXJVj2dgx/FHz+AL0KkEg=</latexit><latexit sha1_base64="Z38XTcs8k8aFZlgJ0FhgA/w97O0=">AB83icbVC7SgNBFL0bXzG+opY2i0GwCrM2prAI2FhGMA9IlmV2MpsMmZld5iGJb9hY6GIrT9j5984SbQxAMXDufcy73xBln2iD07ZU2Nre2d8q7lb39g8Oj6vFJR6dWEdomKU9VL8aciZp2zDaS9TFIuY0248uZ373UeqNEvlg5lmNBR4JFnCDZOGgy0FVH+FLEbNIuqNVRHC/jrJChIDQq0ourXYJgSK6g0hGOt+wHKTJhjZRjhdFYZWE0zTCZ4RPuOSiyoDvPFzTP/wilDP0mVK2n8hfp7IsdC6mIXafAZqxXvbn4n9e3JmEOZOZNVS5aLEct+k/jwAf8gUJYZPHcFEMXerT8ZYWJcTBUXQrD68jrpXNUDVA/uUa3ZKOIowxmcwyUEcA1NuIMWtIFABs/wCm+e9V68d+9j2VryiplT+APv8wf+wpGY</latexit><latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit><latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit><latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit><latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit><latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="VmSdpJnIqoPNIfkVFuEH2jJZP4=">AB6HicbZBLSwMxFIXv1FetVatbN8EiuCoZN7pwIbhxWcG2QmcYMmDU0yQx5iGfo3LhQxF/kzn9j+lho64HAxzkJ9+akheDGYvwdVDY2t7Z3qru1vfr+wWHjqN41udOUdWgucv2YEsMEV6xjuRXsdCMyFSwXjq+neW9J6YNz9WDnRQslmSoeMYpsd6KIuNkUj4n/BpPk0YTt/BcaB3CJTRhqXbS+IoGOXWSKUsFMaYf4sLGJdGWU8GmtcgZVhA6JkPW96iIZCYu5ztP0Zl3BijLtT/Korn7+0VJpDETmfqbktiRWc1m5n9Z39nsKi65Kpxli4GZU4gm6NZAWjANaNWTDwQqrnfFdER0YRaX1PNlxCufnkduhetELfCewxVOIFTOIcQLuEG7qANHaBQwAu8wXvgtfgY1FXJVj2dgx/FHz+AL0KkEg=</latexit><latexit sha1_base64="VmSdpJnIqoPNIfkVFuEH2jJZP4=">AB6HicbZBLSwMxFIXv1FetVatbN8EiuCoZN7pwIbhxWcG2QmcYMmDU0yQx5iGfo3LhQxF/kzn9j+lho64HAxzkJ9+akheDGYvwdVDY2t7Z3qru1vfr+wWHjqN41udOUdWgucv2YEsMEV6xjuRXsdCMyFSwXjq+neW9J6YNz9WDnRQslmSoeMYpsd6KIuNkUj4n/BpPk0YTt/BcaB3CJTRhqXbS+IoGOXWSKUsFMaYf4sLGJdGWU8GmtcgZVhA6JkPW96iIZCYu5ztP0Zl3BijLtT/Korn7+0VJpDETmfqbktiRWc1m5n9Z39nsKi65Kpxli4GZU4gm6NZAWjANaNWTDwQqrnfFdER0YRaX1PNlxCufnkduhetELfCewxVOIFTOIcQLuEG7qANHaBQwAu8wXvgtfgY1FXJVj2dgx/FHz+AL0KkEg=</latexit><latexit sha1_base64="Z38XTcs8k8aFZlgJ0FhgA/w97O0=">AB83icbVC7SgNBFL0bXzG+opY2i0GwCrM2prAI2FhGMA9IlmV2MpsMmZld5iGJb9hY6GIrT9j5984SbQxAMXDufcy73xBln2iD07ZU2Nre2d8q7lb39g8Oj6vFJR6dWEdomKU9VL8aciZp2zDaS9TFIuY0248uZ373UeqNEvlg5lmNBR4JFnCDZOGgy0FVH+FLEbNIuqNVRHC/jrJChIDQq0ourXYJgSK6g0hGOt+wHKTJhjZRjhdFYZWE0zTCZ4RPuOSiyoDvPFzTP/wilDP0mVK2n8hfp7IsdC6mIXafAZqxXvbn4n9e3JmEOZOZNVS5aLEct+k/jwAf8gUJYZPHcFEMXerT8ZYWJcTBUXQrD68jrpXNUDVA/uUa3ZKOIowxmcwyUEcA1NuIMWtIFABs/wCm+e9V68d+9j2VryiplT+APv8wf+wpGY</latexit><latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit><latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit><latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit><latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit><latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit><latexit sha1_base64="vVRZ8mRNWV13Ovzo42xIJqc1ajo=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0RzMFDwIvHCOYB2WZnUySITOzyzEsOQ3vHhQxKs/482/cZLsQRMLGoqbrq7kowzbXz/21tb39jc2i7tlHf39g8OK0fHbZ1aRWiLpDxV3QRrypmkLcMp91MUSwSTjvJ+Hbmdx6p0iyVD2aS0UjgoWQDRrBxUhqK+L8KWY3/jSuVP2aPwdaJUFBqlCgGVe+wn5KrKDSEI617gV+ZqIcK8MIp9NyaDXNMBnjIe05KrGgOsrnN0/RuVP6aJAqV9Kgufp7IsdC64lIXKfAZqSXvZn4n9ezZlCPciYza6gki0UDy5FJ0SwA1GeKEsMnjmCimLsVkRFWmBgXU9mFECy/vEral7XArwX3V9VGvYijBKdwBhcQwDU04A6a0AICGTzDK7x51nvx3r2PReuaV8ycwB94nz8AEZGc</latexit>

𝛽Rxi + X

xi≥0

<latexit sha1_base64="50mNsqgKCkaZ+04s7PjI0dVaZw=">AB+XicbVBNS8NAEN3Ur1q/oh69LBZBEoigj0WvHisYFuhDWGznbRLdzdxd1Msof/EiwdFvPpPvPlv3LY5aOuDgcd7M8zMi1LOtPG8b6e0tr6xuVXeruzs7u0fuIdHbZ1kikKLJjxRDxHRwJmElmGw0OqgIiIQyca3cz8zhiUZom8N5MUAkEGksWMEmOl0HUvejoTYf4Ust4AHr1p6Fa9mjcHXiV+QaqoQDN0v3r9hGYCpKGcaN31vdQEOVGUQ7TSi/TkBI6IgPoWiqJAB3k8un+MwqfRwnypY0eK7+nsiJ0HoiItspiBnqZW8m/ud1MxPXg5zJNDMg6WJRnHFsEjyLAfeZAmr4xBJCFbO3YjokilBjw6rYEPzl1dJ+7LmezX/7qraqBdxlNEJOkXnyEfXqIFuURO1EVj9Ixe0ZuTOy/Ou/OxaC05xcwx+gPn8wdxDZN9</latexit><latexit sha1_base64="50mNsqgKCkaZ+04s7PjI0dVaZw=">AB+XicbVBNS8NAEN3Ur1q/oh69LBZBEoigj0WvHisYFuhDWGznbRLdzdxd1Msof/EiwdFvPpPvPlv3LY5aOuDgcd7M8zMi1LOtPG8b6e0tr6xuVXeruzs7u0fuIdHbZ1kikKLJjxRDxHRwJmElmGw0OqgIiIQyca3cz8zhiUZom8N5MUAkEGksWMEmOl0HUvejoTYf4Ust4AHr1p6Fa9mjcHXiV+QaqoQDN0v3r9hGYCpKGcaN31vdQEOVGUQ7TSi/TkBI6IgPoWiqJAB3k8un+MwqfRwnypY0eK7+nsiJ0HoiItspiBnqZW8m/ud1MxPXg5zJNDMg6WJRnHFsEjyLAfeZAmr4xBJCFbO3YjokilBjw6rYEPzl1dJ+7LmezX/7qraqBdxlNEJOkXnyEfXqIFuURO1EVj9Ixe0ZuTOy/Ou/OxaC05xcwx+gPn8wdxDZN9</latexit><latexit sha1_base64="50mNsqgKCkaZ+04s7PjI0dVaZw=">AB+XicbVBNS8NAEN3Ur1q/oh69LBZBEoigj0WvHisYFuhDWGznbRLdzdxd1Msof/EiwdFvPpPvPlv3LY5aOuDgcd7M8zMi1LOtPG8b6e0tr6xuVXeruzs7u0fuIdHbZ1kikKLJjxRDxHRwJmElmGw0OqgIiIQyca3cz8zhiUZom8N5MUAkEGksWMEmOl0HUvejoTYf4Ust4AHr1p6Fa9mjcHXiV+QaqoQDN0v3r9hGYCpKGcaN31vdQEOVGUQ7TSi/TkBI6IgPoWiqJAB3k8un+MwqfRwnypY0eK7+nsiJ0HoiItspiBnqZW8m/ud1MxPXg5zJNDMg6WJRnHFsEjyLAfeZAmr4xBJCFbO3YjokilBjw6rYEPzl1dJ+7LmezX/7qraqBdxlNEJOkXnyEfXqIFuURO1EVj9Ixe0ZuTOy/Ou/OxaC05xcwx+gPn8wdxDZN9</latexit><latexit sha1_base64="50mNsqgKCkaZ+04s7PjI0dVaZw=">AB+XicbVBNS8NAEN3Ur1q/oh69LBZBEoigj0WvHisYFuhDWGznbRLdzdxd1Msof/EiwdFvPpPvPlv3LY5aOuDgcd7M8zMi1LOtPG8b6e0tr6xuVXeruzs7u0fuIdHbZ1kikKLJjxRDxHRwJmElmGw0OqgIiIQyca3cz8zhiUZom8N5MUAkEGksWMEmOl0HUvejoTYf4Ust4AHr1p6Fa9mjcHXiV+QaqoQDN0v3r9hGYCpKGcaN31vdQEOVGUQ7TSi/TkBI6IgPoWiqJAB3k8un+MwqfRwnypY0eK7+nsiJ0HoiItspiBnqZW8m/ud1MxPXg5zJNDMg6WJRnHFsEjyLAfeZAmr4xBJCFbO3YjokilBjw6rYEPzl1dJ+7LmezX/7qraqBdxlNEJOkXnyEfXqIFuURO1EVj9Ixe0ZuTOy/Ou/OxaC05xcwx+gPn8wdxDZN9</latexit>

𝛽xi ≈ X

xi≥0

<latexit sha1_base64="OmUN7o15gBwrSbQz2UYW17u/s0s=">ACAHicbVDLSsNAFJ3UV62vqAsXbgaL4KokIthlwY3LCvYBTQiT6aQdOo84M5GW0I2/4saFIm79DHf+jdM2C209cOFwzr3ce0+cMqN5307pbX1jc2t8nZlZ3dv/8A9PGprmSlMWlgyqbox0oRQVqGka6qSKIx4x04tHNzO8EqWpFPdmkpKQo4GgCcXIWClyTwKUpkqOYaAzHuXjiAYD8uBNI7fq1bw54CrxC1IFBZqR+xX0Jc4EQYzpHXP91IT5kgZihmZVoJMkxThERqQnqUCcaLDfP7AFJ5bpQ8TqWwJA+fq74kca0nPLadHJmhXvZm4n9eLzNJPcypSDNDBF4sSjIGjYSzNGCfKoINm1iCsKL2VoiHSCFsbGYVG4K/PIqaV/WfK/m31VG/UijI4BWfgAvjgGjTALWiCFsBgCp7BK3hznpwX5935WLSWnGLmGPyB8/kDGM+Wrg=</latexit><latexit sha1_base64="OmUN7o15gBwrSbQz2UYW17u/s0s=">ACAHicbVDLSsNAFJ3UV62vqAsXbgaL4KokIthlwY3LCvYBTQiT6aQdOo84M5GW0I2/4saFIm79DHf+jdM2C209cOFwzr3ce0+cMqN5307pbX1jc2t8nZlZ3dv/8A9PGprmSlMWlgyqbox0oRQVqGka6qSKIx4x04tHNzO8EqWpFPdmkpKQo4GgCcXIWClyTwKUpkqOYaAzHuXjiAYD8uBNI7fq1bw54CrxC1IFBZqR+xX0Jc4EQYzpHXP91IT5kgZihmZVoJMkxThERqQnqUCcaLDfP7AFJ5bpQ8TqWwJA+fq74kca0nPLadHJmhXvZm4n9eLzNJPcypSDNDBF4sSjIGjYSzNGCfKoINm1iCsKL2VoiHSCFsbGYVG4K/PIqaV/WfK/m31VG/UijI4BWfgAvjgGjTALWiCFsBgCp7BK3hznpwX5935WLSWnGLmGPyB8/kDGM+Wrg=</latexit><latexit sha1_base64="OmUN7o15gBwrSbQz2UYW17u/s0s=">ACAHicbVDLSsNAFJ3UV62vqAsXbgaL4KokIthlwY3LCvYBTQiT6aQdOo84M5GW0I2/4saFIm79DHf+jdM2C209cOFwzr3ce0+cMqN5307pbX1jc2t8nZlZ3dv/8A9PGprmSlMWlgyqbox0oRQVqGka6qSKIx4x04tHNzO8EqWpFPdmkpKQo4GgCcXIWClyTwKUpkqOYaAzHuXjiAYD8uBNI7fq1bw54CrxC1IFBZqR+xX0Jc4EQYzpHXP91IT5kgZihmZVoJMkxThERqQnqUCcaLDfP7AFJ5bpQ8TqWwJA+fq74kca0nPLadHJmhXvZm4n9eLzNJPcypSDNDBF4sSjIGjYSzNGCfKoINm1iCsKL2VoiHSCFsbGYVG4K/PIqaV/WfK/m31VG/UijI4BWfgAvjgGjTALWiCFsBgCp7BK3hznpwX5935WLSWnGLmGPyB8/kDGM+Wrg=</latexit><latexit sha1_base64="OmUN7o15gBwrSbQz2UYW17u/s0s=">ACAHicbVDLSsNAFJ3UV62vqAsXbgaL4KokIthlwY3LCvYBTQiT6aQdOo84M5GW0I2/4saFIm79DHf+jdM2C209cOFwzr3ce0+cMqN5307pbX1jc2t8nZlZ3dv/8A9PGprmSlMWlgyqbox0oRQVqGka6qSKIx4x04tHNzO8EqWpFPdmkpKQo4GgCcXIWClyTwKUpkqOYaAzHuXjiAYD8uBNI7fq1bw54CrxC1IFBZqR+xX0Jc4EQYzpHXP91IT5kgZihmZVoJMkxThERqQnqUCcaLDfP7AFJ5bpQ8TqWwJA+fq74kca0nPLadHJmhXvZm4n9eLzNJPcypSDNDBF4sSjIGjYSzNGCfKoINm1iCsKL2VoiHSCFsbGYVG4K/PIqaV/WfK/m31VG/UijI4BWfgAvjgGjTALWiCFsBgCp7BK3hznpwX5935WLSWnGLmGPyB8/kDGM+Wrg=</latexit>

𝛽Rxi Plug the value from previous slide and recurse for K epochs

(assume R >> 1)

slide-15
SLIDE 15

Conclusion

  • In this work, we close the gap in convergence rates of SGDo.
  • We discovered an interesting phenomenon :

SGDo converges faster for strongly convex quadratics than general strongly convex smooth functions.

Future Work

  • Do there exist “optimal” permutations? Distribution of convergence rates for permutations.
  • Can these analyses be extended to algorithms that compress gradients?
  • Can we analyze convergence for “system-friendly” shuffling schemes?