Nonconvex Variance Reduced Optimization with Arbitrary Sampling - - PowerPoint PPT Presentation

nonconvex variance reduced optimization with arbitrary
SMART_READER_LITE
LIVE PREVIEW

Nonconvex Variance Reduced Optimization with Arbitrary Sampling - - PowerPoint PPT Presentation

Nonconvex Variance Reduced Optimization with Arbitrary Sampling Samuel Horvth Peter Richtrik Empirical Risk Minimization n x R d f ( x ) := 1 X min f i ( x ) n i =1 Empirical Risk Minimization n x R d f ( x ) := 1 X min f i (


slide-1
SLIDE 1

Nonconvex Variance Reduced Optimization with Arbitrary Sampling

Samuel Horváth Peter Richtárik

slide-2
SLIDE 2

Empirical Risk Minimization

min

x∈Rd f(x) := 1

n

n

X

i=1

fi(x)

slide-3
SLIDE 3

Empirical Risk Minimization

min

x∈Rd f(x) := 1

n

n

X

i=1

fi(x)

n is big

slide-4
SLIDE 4

Empirical Risk Minimization

min

x∈Rd f(x) := 1

n

n

X

i=1

fi(x)

n is big

krfi(x) rfi(y)k  Likx yk

non-convex, Li-smooth

slide-5
SLIDE 5

Baseline Variance Reduced SGD Methods

SVRG SAGA SARAH

Nguyen, Liu, Scheinberg & Takáč ICML 2017 Defazio, Bach & Lacoste-Julien NIPS 2014 Johnson & Zhang NIPS 2013

slide-6
SLIDE 6

Baseline Variance Reduced SGD Methods

SVRG SAGA SARAH

Nguyen, Liu, Scheinberg & Takáč ICML 2017 Defazio, Bach & Lacoste-Julien NIPS 2014 Johnson & Zhang NIPS 2013

Uniform sampling Uniform sampling Uniform sampling

slide-7
SLIDE 7

Baseline Variance Reduced SGD Methods–Mini-batch

SVRG SAGA SARAH

Nguyen, Liu, Scheinberg & Takáč 2017 Reddi, Hefny, Sra, Poczos, Smola CDC 2016 Konečný & Richtárik FAMS 2017

slide-8
SLIDE 8

Baseline Variance Reduced SGD Methods–Mini-batch

SVRG SAGA SARAH

Nguyen, Liu, Scheinberg & Takáč 2017 Reddi, Hefny, Sra, Poczos, Smola CDC 2016 Konečný & Richtárik FAMS 2017

Uniform sampling Uniform sampling Uniform sampling Mini-batch size

slide-9
SLIDE 9

Contributions

  • Analysis of SVRG, SAGA and SARAH in the arbitrary sampling paradigm
  • Construction of optimal minibatch sampling
slide-10
SLIDE 10

Contributions

  • Analysis of SVRG, SAGA and SARAH in the arbitrary sampling paradigm
  • Construction of optimal minibatch sampling

Richtárik & Takáč (OL 2016; arXiv 2013) Qu, Richtárik & Zhang (NIPS 2015) Qu & Richtárik (COAP 2016) Chambolle, Ehrhardt, Richtárik & Schoenlieb (SIOPT 2018) Hanzely & Richtárik (AISTATS 2019) Qian, Qu & Richtárik (ICML 2019) Gower, Loizou, Qian, Sailanbayev, Shulgin & Richtárik (ICML 2019)

slide-11
SLIDE 11

Contributions

  • Analysis of SVRG, SAGA and SARAH in the arbitrary sampling paradigm
  • Construction of optimal minibatch sampling

Richtárik & Takáč (OL 2016; arXiv 2013) Qu, Richtárik & Zhang (NIPS 2015) Qu & Richtárik (COAP 2016) Chambolle, Ehrhardt, Richtárik & Schoenlieb (SIOPT 2018) Hanzely & Richtárik (AISTATS 2019) Qian, Qu & Richtárik (ICML 2019) Gower, Loizou, Qian, Sailanbayev, Shulgin & Richtárik (ICML 2019)

First optimal/importance sampling for minibatches!

slide-12
SLIDE 12

Data Sampling (i.e., Mini-batching) Mechanisms

Sampling: a random subset of {1, 2, … , n}

Pij := Prob({i, j} ⊆ S)

pi := Prob({i} ⊆ S) = Pii

Probability matrix associated with sampling Probability vector associated with sampling P ∈ Rn×n

<latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit>

S

<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>

p ∈ Rn

<latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit>

Proper sampling: pi > 0

<latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit>

for all i = 1, 2, . . . , n

<latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit>

S

<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>
slide-13
SLIDE 13

Data Sampling (i.e., Mini-batching) Mechanisms

Sampling: a random subset of {1, 2, … , n}

A sampling is uniquely defined by assigning probabilities to all 2n subsets of {1, 2, ... , n}

Pij := Prob({i, j} ⊆ S)

pi := Prob({i} ⊆ S) = Pii

Probability matrix associated with sampling Probability vector associated with sampling P ∈ Rn×n

<latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit>

S

<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>

p ∈ Rn

<latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit>

Proper sampling: pi > 0

<latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit>

for all i = 1, 2, . . . , n

<latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit>

S

<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>
slide-14
SLIDE 14

Data Sampling (i.e., Mini-batching) Mechanisms

Sampling: a random subset of {1, 2, … , n}

A sampling is uniquely defined by assigning probabilities to all 2n subsets of {1, 2, ... , n}

Pij := Prob({i, j} ⊆ S)

pi := Prob({i} ⊆ S) = Pii

Probability matrix associated with sampling Probability vector associated with sampling P ∈ Rn×n

<latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit>

S

<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>

p ∈ Rn

<latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit>

Proper sampling: pi > 0

<latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit>

for all i = 1, 2, . . . , n

<latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit>

S

<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>

Standard sampling:

Examples

S = {i} with probability 1

n for all i = 1, 2, . . . , n

<latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit><latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit><latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit><latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit>

S = C with probability

1

(

n b)

for all C ⊂ {1, 2, . . . , n} such that |C| = b

<latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit><latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit><latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit><latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit>

Standard mini-batch sampling:

slide-15
SLIDE 15

Data Sampling (i.e., Mini-batching) Mechanisms

Sampling: a random subset of {1, 2, … , n}

A sampling is uniquely defined by assigning probabilities to all 2n subsets of {1, 2, ... , n}

Pij := Prob({i, j} ⊆ S)

pi := Prob({i} ⊆ S) = Pii

Probability matrix associated with sampling Probability vector associated with sampling P ∈ Rn×n

<latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit>

S

<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>

p ∈ Rn

<latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit>

Proper sampling: pi > 0

<latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit>

for all i = 1, 2, . . . , n

<latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit>

S

<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>

Arbitrary sampling paradigm = perform iteration complexity analysis for any proper sampling

Standard sampling:

Examples

S = {i} with probability 1

n for all i = 1, 2, . . . , n

<latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit><latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit><latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit><latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit>

S = C with probability

1

(

n b)

for all C ⊂ {1, 2, . . . , n} such that |C| = b

<latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit><latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit><latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit><latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit>

Standard mini-batch sampling:

slide-16
SLIDE 16

From Standard Sampling to Arbitrary Sampling

SVRG with Arbitrary Sampling x+ = x η X

i∈S

1 npi (rfi(x) rfi(ˆ x)) + rf(ˆ x) !

<latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit>
slide-17
SLIDE 17

From Standard Sampling to Arbitrary Sampling

Unbiased estimator of the gradient

SVRG with Arbitrary Sampling x+ = x η X

i∈S

1 npi (rfi(x) rfi(ˆ x)) + rf(ˆ x) !

<latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit>
slide-18
SLIDE 18

From Standard Sampling to Arbitrary Sampling

Unbiased estimator of the gradient

SVRG with Arbitrary Sampling x+ = x η X

i∈S

1 npi (rfi(x) rfi(ˆ x)) + rf(ˆ x) !

<latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit>

Arbitrary sampling

pi = 1

n for all i

<latexit sha1_base64="LmnDBt30CO6XQ28Rzf/6r/4U7Hs=">ACHicbVC7TsMwFHV4lvIKMDJg0SAxVUkXWJAqWBiLRB9SG0WO67RWHTuyHaQqysjCr7AwgBArn8DG3+CmGaDlTEfn3Kt7zwkTRpV23W9rZXVtfWOzslXd3tnd27cPDjtKpBKTNhZMyF6IFGUk7ampFeIgmKQ0a64eRm5ncfiFRU8Hs9TYgfoxGnEcVIGymwT5wkoPAKDiKJcOblGc8dGAkJEWPQoU5g19y6WwAuE68kNVCiFdhfg6HAaUy4xgwp1fcRPsZkpiRvLqIFUkQXiCRqRvKEcxUX5WBMnhmVGxflIcA0L9fdGhmKlpnFoJmOkx2rRm4n/ef1UR5d+RnmSasLx/FCUMqgFnLUCh1QSrNnUEIQlNb9CPEamEm26q5oSvMXIy6TqHtu3btr1JrXZR0VcAxOwTnwAVoglvQAm2AwSN4Bq/gzXqyXqx362M+umKVO0fgD6zPH+Iul/Q=</latexit><latexit sha1_base64="LmnDBt30CO6XQ28Rzf/6r/4U7Hs=">ACHicbVC7TsMwFHV4lvIKMDJg0SAxVUkXWJAqWBiLRB9SG0WO67RWHTuyHaQqysjCr7AwgBArn8DG3+CmGaDlTEfn3Kt7zwkTRpV23W9rZXVtfWOzslXd3tnd27cPDjtKpBKTNhZMyF6IFGUk7ampFeIgmKQ0a64eRm5ncfiFRU8Hs9TYgfoxGnEcVIGymwT5wkoPAKDiKJcOblGc8dGAkJEWPQoU5g19y6WwAuE68kNVCiFdhfg6HAaUy4xgwp1fcRPsZkpiRvLqIFUkQXiCRqRvKEcxUX5WBMnhmVGxflIcA0L9fdGhmKlpnFoJmOkx2rRm4n/ef1UR5d+RnmSasLx/FCUMqgFnLUCh1QSrNnUEIQlNb9CPEamEm26q5oSvMXIy6TqHtu3btr1JrXZR0VcAxOwTnwAVoglvQAm2AwSN4Bq/gzXqyXqx362M+umKVO0fgD6zPH+Iul/Q=</latexit><latexit sha1_base64="LmnDBt30CO6XQ28Rzf/6r/4U7Hs=">ACHicbVC7TsMwFHV4lvIKMDJg0SAxVUkXWJAqWBiLRB9SG0WO67RWHTuyHaQqysjCr7AwgBArn8DG3+CmGaDlTEfn3Kt7zwkTRpV23W9rZXVtfWOzslXd3tnd27cPDjtKpBKTNhZMyF6IFGUk7ampFeIgmKQ0a64eRm5ncfiFRU8Hs9TYgfoxGnEcVIGymwT5wkoPAKDiKJcOblGc8dGAkJEWPQoU5g19y6WwAuE68kNVCiFdhfg6HAaUy4xgwp1fcRPsZkpiRvLqIFUkQXiCRqRvKEcxUX5WBMnhmVGxflIcA0L9fdGhmKlpnFoJmOkx2rRm4n/ef1UR5d+RnmSasLx/FCUMqgFnLUCh1QSrNnUEIQlNb9CPEamEm26q5oSvMXIy6TqHtu3btr1JrXZR0VcAxOwTnwAVoglvQAm2AwSN4Bq/gzXqyXqx362M+umKVO0fgD6zPH+Iul/Q=</latexit><latexit sha1_base64="LmnDBt30CO6XQ28Rzf/6r/4U7Hs=">ACHicbVC7TsMwFHV4lvIKMDJg0SAxVUkXWJAqWBiLRB9SG0WO67RWHTuyHaQqysjCr7AwgBArn8DG3+CmGaDlTEfn3Kt7zwkTRpV23W9rZXVtfWOzslXd3tnd27cPDjtKpBKTNhZMyF6IFGUk7ampFeIgmKQ0a64eRm5ncfiFRU8Hs9TYgfoxGnEcVIGymwT5wkoPAKDiKJcOblGc8dGAkJEWPQoU5g19y6WwAuE68kNVCiFdhfg6HAaUy4xgwp1fcRPsZkpiRvLqIFUkQXiCRqRvKEcxUX5WBMnhmVGxflIcA0L9fdGhmKlpnFoJmOkx2rRm4n/ef1UR5d+RnmSasLx/FCUMqgFnLUCh1QSrNnUEIQlNb9CPEamEm26q5oSvMXIy6TqHtu3btr1JrXZR0VcAxOwTnwAVoglvQAm2AwSN4Bq/gzXqyXqx362M+umKVO0fgD6zPH+Iul/Q=</latexit>

Standard sampling:

slide-19
SLIDE 19

Convergence Rate I

O ✓ n + ↵n2/3 ✏ ◆

<latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="Y91LCEnI6UTKSgw/m5WIwSPmBU=">ACJ3icbZBaxRBEIVrYqLJurq1UuTICQE1pl40KPgJZ6M4CaB7XGp6a3ZbdLTM3TXCEszv8lLfkgBz0o4h+xd7MHTfKg4fFeF139FY3RntP0e7L2YH3j4aPNrd72zuMnT/vPdk593TpFQ1Wb2p0X6MloS0PWbOi8cYRVYeisuHi/6M+kvO6tp953lBe4dTqUivkGI37H+QoSIVGfOykoZL3rTgUMVkMBkeTqJpZtgJWTpUwX4JR69ed12Q1HhtatJp6czPpD5uL+XDtKlxF2TrcwerHQy7l/JSa3aiwrg96PsrThPKBjrQx1Pdl6alBd4JRG0VqsyOdhuVknXsZkIsraxWNZLN/JwJW3s+rIt6skGf+drcI7+tGLZdv86Bt0zJZdfNQ2RrBtVjwExPtSLGZR4PK6birUDOMbDhS7kUI2e0v3zWnR4MsHWSfUtiEF7AL+5DBG3gHx3ACQ1DwDa7hJ/xKLpMfye8bXGvJitz+E/Jn7/Icqm0</latexit><latexit sha1_base64="Y91LCEnI6UTKSgw/m5WIwSPmBU=">ACJ3icbZBaxRBEIVrYqLJurq1UuTICQE1pl40KPgJZ6M4CaB7XGp6a3ZbdLTM3TXCEszv8lLfkgBz0o4h+xd7MHTfKg4fFeF139FY3RntP0e7L2YH3j4aPNrd72zuMnT/vPdk593TpFQ1Wb2p0X6MloS0PWbOi8cYRVYeisuHi/6M+kvO6tp953lBe4dTqUivkGI37H+QoSIVGfOykoZL3rTgUMVkMBkeTqJpZtgJWTpUwX4JR69ed12Q1HhtatJp6czPpD5uL+XDtKlxF2TrcwerHQy7l/JSa3aiwrg96PsrThPKBjrQx1Pdl6alBd4JRG0VqsyOdhuVknXsZkIsraxWNZLN/JwJW3s+rIt6skGf+drcI7+tGLZdv86Bt0zJZdfNQ2RrBtVjwExPtSLGZR4PK6birUDOMbDhS7kUI2e0v3zWnR4MsHWSfUtiEF7AL+5DBG3gHx3ACQ1DwDa7hJ/xKLpMfye8bXGvJitz+E/Jn7/Icqm0</latexit><latexit sha1_base64="qK24elSUrYa+kftlV+eYmZ3WCdo=">ACMnicbVDBShxBFOzRJrVxFWPuTRZAkpgM2MO8SjxEk8ayKqwPS5vet/sNvb0DN1vhKWZb8rFLxE8xIMiXvMR6V3kGgKGoqV7zXlVaOYrjX9HC4ouXr5aWX7dWVt+8XWuvbxy7srYSe7LUpT3NwKFWBnukSONpZRGKTONJdr4/9U8u0DpVmh80qTAtYGRUriRQkAbtA9H3QoLmh43QmNOW4R95UKZBb3HYCNDVGBoucgvSmzO/8+lz03iBlVO6NI2wajSmbZEO2p24G8/An5NkTjpsjqNB+0oMS1kXaEhqcK6fxBWlHiwpqbFpidphBfIcRtgP1ECBLvWzyxr+IShDnpc2PEN8pv6d8FA4NymyMFkAjd1Tbyr+z+vXlO+mXpmqJjTycVFea04ln/bHh8qiJD0JBKRV4VYuxC6odByK5SQP3yc3K803ibvI97ux9ndexzN6x92yLJewL2Pf2BHrMcl+smt2y+6iy+gmuo8eHkcXonlmk/2D6PcfHFGrTQ=</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit>
slide-20
SLIDE 20

Convergence Rate I

O ✓ n + ↵n2/3 ✏ ◆

<latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="Y91LCEnI6UTKSgw/m5WIwSPmBU=">ACJ3icbZBaxRBEIVrYqLJurq1UuTICQE1pl40KPgJZ6M4CaB7XGp6a3ZbdLTM3TXCEszv8lLfkgBz0o4h+xd7MHTfKg4fFeF139FY3RntP0e7L2YH3j4aPNrd72zuMnT/vPdk593TpFQ1Wb2p0X6MloS0PWbOi8cYRVYeisuHi/6M+kvO6tp953lBe4dTqUivkGI37H+QoSIVGfOykoZL3rTgUMVkMBkeTqJpZtgJWTpUwX4JR69ed12Q1HhtatJp6czPpD5uL+XDtKlxF2TrcwerHQy7l/JSa3aiwrg96PsrThPKBjrQx1Pdl6alBd4JRG0VqsyOdhuVknXsZkIsraxWNZLN/JwJW3s+rIt6skGf+drcI7+tGLZdv86Bt0zJZdfNQ2RrBtVjwExPtSLGZR4PK6birUDOMbDhS7kUI2e0v3zWnR4MsHWSfUtiEF7AL+5DBG3gHx3ACQ1DwDa7hJ/xKLpMfye8bXGvJitz+E/Jn7/Icqm0</latexit><latexit sha1_base64="Y91LCEnI6UTKSgw/m5WIwSPmBU=">ACJ3icbZBaxRBEIVrYqLJurq1UuTICQE1pl40KPgJZ6M4CaB7XGp6a3ZbdLTM3TXCEszv8lLfkgBz0o4h+xd7MHTfKg4fFeF139FY3RntP0e7L2YH3j4aPNrd72zuMnT/vPdk593TpFQ1Wb2p0X6MloS0PWbOi8cYRVYeisuHi/6M+kvO6tp953lBe4dTqUivkGI37H+QoSIVGfOykoZL3rTgUMVkMBkeTqJpZtgJWTpUwX4JR69ed12Q1HhtatJp6czPpD5uL+XDtKlxF2TrcwerHQy7l/JSa3aiwrg96PsrThPKBjrQx1Pdl6alBd4JRG0VqsyOdhuVknXsZkIsraxWNZLN/JwJW3s+rIt6skGf+drcI7+tGLZdv86Bt0zJZdfNQ2RrBtVjwExPtSLGZR4PK6birUDOMbDhS7kUI2e0v3zWnR4MsHWSfUtiEF7AL+5DBG3gHx3ACQ1DwDa7hJ/xKLpMfye8bXGvJitz+E/Jn7/Icqm0</latexit><latexit sha1_base64="qK24elSUrYa+kftlV+eYmZ3WCdo=">ACMnicbVDBShxBFOzRJrVxFWPuTRZAkpgM2MO8SjxEk8ayKqwPS5vet/sNvb0DN1vhKWZb8rFLxE8xIMiXvMR6V3kGgKGoqV7zXlVaOYrjX9HC4ouXr5aWX7dWVt+8XWuvbxy7srYSe7LUpT3NwKFWBnukSONpZRGKTONJdr4/9U8u0DpVmh80qTAtYGRUriRQkAbtA9H3QoLmh43QmNOW4R95UKZBb3HYCNDVGBoucgvSmzO/8+lz03iBlVO6NI2wajSmbZEO2p24G8/An5NkTjpsjqNB+0oMS1kXaEhqcK6fxBWlHiwpqbFpidphBfIcRtgP1ECBLvWzyxr+IShDnpc2PEN8pv6d8FA4NymyMFkAjd1Tbyr+z+vXlO+mXpmqJjTycVFea04ln/bHh8qiJD0JBKRV4VYuxC6odByK5SQP3yc3K803ibvI97ux9ndexzN6x92yLJewL2Pf2BHrMcl+smt2y+6iy+gmuo8eHkcXonlmk/2D6PcfHFGrTQ=</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit>

# data points

Ekrf(x)k2  ✏

<latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit><latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit><latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit><latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit>
slide-21
SLIDE 21

Convergence Rate I

O ✓ n + ↵n2/3 ✏ ◆

<latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQvEKMXyw=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odBu3wMYA6nMFXEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="Y91LCEnI6UTKSgw/m5WIwSPmBU=">ACJ3icbZBaxRBEIVrYqLJurq1UuTICQE1pl40KPgJZ6M4CaB7XGp6a3ZbdLTM3TXCEszv8lLfkgBz0o4h+xd7MHTfKg4fFeF139FY3RntP0e7L2YH3j4aPNrd72zuMnT/vPdk593TpFQ1Wb2p0X6MloS0PWbOi8cYRVYeisuHi/6M+kvO6tp953lBe4dTqUivkGI37H+QoSIVGfOykoZL3rTgUMVkMBkeTqJpZtgJWTpUwX4JR69ed12Q1HhtatJp6czPpD5uL+XDtKlxF2TrcwerHQy7l/JSa3aiwrg96PsrThPKBjrQx1Pdl6alBd4JRG0VqsyOdhuVknXsZkIsraxWNZLN/JwJW3s+rIt6skGf+drcI7+tGLZdv86Bt0zJZdfNQ2RrBtVjwExPtSLGZR4PK6birUDOMbDhS7kUI2e0v3zWnR4MsHWSfUtiEF7AL+5DBG3gHx3ACQ1DwDa7hJ/xKLpMfye8bXGvJitz+E/Jn7/Icqm0</latexit><latexit sha1_base64="Y91LCEnI6UTKSgw/m5WIwSPmBU=">ACJ3icbZBaxRBEIVrYqLJurq1UuTICQE1pl40KPgJZ6M4CaB7XGp6a3ZbdLTM3TXCEszv8lLfkgBz0o4h+xd7MHTfKg4fFeF139FY3RntP0e7L2YH3j4aPNrd72zuMnT/vPdk593TpFQ1Wb2p0X6MloS0PWbOi8cYRVYeisuHi/6M+kvO6tp953lBe4dTqUivkGI37H+QoSIVGfOykoZL3rTgUMVkMBkeTqJpZtgJWTpUwX4JR69ed12Q1HhtatJp6czPpD5uL+XDtKlxF2TrcwerHQy7l/JSa3aiwrg96PsrThPKBjrQx1Pdl6alBd4JRG0VqsyOdhuVknXsZkIsraxWNZLN/JwJW3s+rIt6skGf+drcI7+tGLZdv86Bt0zJZdfNQ2RrBtVjwExPtSLGZR4PK6birUDOMbDhS7kUI2e0v3zWnR4MsHWSfUtiEF7AL+5DBG3gHx3ACQ1DwDa7hJ/xKLpMfye8bXGvJitz+E/Jn7/Icqm0</latexit><latexit sha1_base64="qK24elSUrYa+kftlV+eYmZ3WCdo=">ACMnicbVDBShxBFOzRJrVxFWPuTRZAkpgM2MO8SjxEk8ayKqwPS5vet/sNvb0DN1vhKWZb8rFLxE8xIMiXvMR6V3kGgKGoqV7zXlVaOYrjX9HC4ouXr5aWX7dWVt+8XWuvbxy7srYSe7LUpT3NwKFWBnukSONpZRGKTONJdr4/9U8u0DpVmh80qTAtYGRUriRQkAbtA9H3QoLmh43QmNOW4R95UKZBb3HYCNDVGBoucgvSmzO/8+lz03iBlVO6NI2wajSmbZEO2p24G8/An5NkTjpsjqNB+0oMS1kXaEhqcK6fxBWlHiwpqbFpidphBfIcRtgP1ECBLvWzyxr+IShDnpc2PEN8pv6d8FA4NymyMFkAjd1Tbyr+z+vXlO+mXpmqJjTycVFea04ln/bHh8qiJD0JBKRV4VYuxC6odByK5SQP3yc3K803ibvI97ux9ndexzN6x92yLJewL2Pf2BHrMcl+smt2y+6iy+gmuo8eHkcXonlmk/2D6PcfHFGrTQ=</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit><latexit sha1_base64="+hy0iAcgGb4aKCmWh67HzW4ZvW0=">ACMnicbVBNaxsxFNQmbZO6H3GTYy+ipBScHfdQ3MyaU9xYU6CVhb81Z+a4totYv0tmDE/qZc8ksKPbSHhJBrfkTkj0ObdEAwzLzhPU1WaeUojn9Ha+uPHj/Z2Hzaevb8xcut9qvtY1fWVuJAlrq0pxk41MrgBRpPK0sQpFpPMnODuf+yQ+0TpXmG80qTAuYGJUrCRSkUfuLGHohQfOjRmjMadfw9zwo86C3OG4E6GoKDRe5BenNd9/78LFpvMDKV2aRlg1mdI7kY7anbgbL8AfkmRFOmyF/qj9U4xLWRdoSGpwbpjEFaUeLCmpsWmJ2mEF8gwmOAzUQIEu9YvLGv42KGOelzY8Q3yh/p3wUDg3K7IwWQBN3X1vLv7PG9aU76VemaomNHK5K81p5LP+NjZVGSngUC0qpwK5dTCN1QaLkVSkjuf/khOe51k7ibfO19g9WdWy1+wN2UJ+8T2WfWZwMm2Tn7xS7ZVXQR/Ymuo5vl6Fq0yuywfxDd3gEc8atP</latexit>

# data points

Ekrf(x)k2  ✏

<latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit><latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit><latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit><latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit>

KEY QUANTITY: DEPENDS ON THE SAMPLING

slide-22
SLIDE 22

Convergence Rate II

α := b ¯ Ln2

n

X

i=1

viL2

i

pi

slide-23
SLIDE 23

Convergence Rate II

α := b ¯ Ln2

n

X

i=1

viL2

i

pi

Pij := Prob({i, j} ⊆ S) pi := Prob({i} ⊆ S) = Pii P pp> Diag(p1v1, p2v2, . . . , pnvn).

Constants satisfying: Expected mini-batch size:

b = E|S|

<latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit><latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit><latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit><latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit>

# data points

¯ L = 1 n

n

X

i=1

Li

<latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit><latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit><latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit><latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit>

krfi(x) rfi(y)k  Likx yk

slide-24
SLIDE 24

Convergence Rate II

α := b ¯ Ln2

n

X

i=1

viL2

i

pi

Optimal rate: minimize over {(vi, pi)}n

i=1

Pij := Prob({i, j} ⊆ S) pi := Prob({i} ⊆ S) = Pii P pp> Diag(p1v1, p2v2, . . . , pnvn).

Constants satisfying: Expected mini-batch size:

b = E|S|

<latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit><latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit><latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit><latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit>

# data points

¯ L = 1 n

n

X

i=1

Li

<latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit><latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit><latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit><latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit>

krfi(x) rfi(y)k  Likx yk

α

<latexit sha1_base64="3cuH/msNH5GL8ZP1kaOEpuPH6OM=">AB73icbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lpjIRwIXMrfswYa9vXN3z4Rc+BM2Fhpj69+x89+4wBUKvmSl/dmMjMvSATXxnW/ncLG5tb2TnG3tLd/cHhUPj5p6zhVlLVoLGLVDVAzwSVrGW4E6yaKYRQI1gkmt3O/8SU5rF8MNOE+RGOJA85RWOlbrWPIhljdVCuDV3AbJOvJxUIEdzUP7qD2OaRkwaKlDrnucmxs9QGU4Fm5X6qWYJ0gmOWM9SiRHTfra4d0YurDIkYaxsSUMW6u+JDCOtp1FgOyM0Y73qzcX/vF5qwms/4zJDZN0uShMBTExmT9PhlwxasTUEqSK21sJHaNCamxEJRuCt/ryOmnXa5b8+7rlcZNHkcRzuAcLsGDK2jAHTShBRQEPMrvDmPzovz7nwsWwtOPnMKf+B8/gBFTI9y</latexit><latexit sha1_base64="3cuH/msNH5GL8ZP1kaOEpuPH6OM=">AB73icbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lpjIRwIXMrfswYa9vXN3z4Rc+BM2Fhpj69+x89+4wBUKvmSl/dmMjMvSATXxnW/ncLG5tb2TnG3tLd/cHhUPj5p6zhVlLVoLGLVDVAzwSVrGW4E6yaKYRQI1gkmt3O/8SU5rF8MNOE+RGOJA85RWOlbrWPIhljdVCuDV3AbJOvJxUIEdzUP7qD2OaRkwaKlDrnucmxs9QGU4Fm5X6qWYJ0gmOWM9SiRHTfra4d0YurDIkYaxsSUMW6u+JDCOtp1FgOyM0Y73qzcX/vF5qwms/4zJDZN0uShMBTExmT9PhlwxasTUEqSK21sJHaNCamxEJRuCt/ryOmnXa5b8+7rlcZNHkcRzuAcLsGDK2jAHTShBRQEPMrvDmPzovz7nwsWwtOPnMKf+B8/gBFTI9y</latexit><latexit sha1_base64="3cuH/msNH5GL8ZP1kaOEpuPH6OM=">AB73icbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lpjIRwIXMrfswYa9vXN3z4Rc+BM2Fhpj69+x89+4wBUKvmSl/dmMjMvSATXxnW/ncLG5tb2TnG3tLd/cHhUPj5p6zhVlLVoLGLVDVAzwSVrGW4E6yaKYRQI1gkmt3O/8SU5rF8MNOE+RGOJA85RWOlbrWPIhljdVCuDV3AbJOvJxUIEdzUP7qD2OaRkwaKlDrnucmxs9QGU4Fm5X6qWYJ0gmOWM9SiRHTfra4d0YurDIkYaxsSUMW6u+JDCOtp1FgOyM0Y73qzcX/vF5qwms/4zJDZN0uShMBTExmT9PhlwxasTUEqSK21sJHaNCamxEJRuCt/ryOmnXa5b8+7rlcZNHkcRzuAcLsGDK2jAHTShBRQEPMrvDmPzovz7nwsWwtOPnMKf+B8/gBFTI9y</latexit><latexit sha1_base64="3cuH/msNH5GL8ZP1kaOEpuPH6OM=">AB73icbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lpjIRwIXMrfswYa9vXN3z4Rc+BM2Fhpj69+x89+4wBUKvmSl/dmMjMvSATXxnW/ncLG5tb2TnG3tLd/cHhUPj5p6zhVlLVoLGLVDVAzwSVrGW4E6yaKYRQI1gkmt3O/8SU5rF8MNOE+RGOJA85RWOlbrWPIhljdVCuDV3AbJOvJxUIEdzUP7qD2OaRkwaKlDrnucmxs9QGU4Fm5X6qWYJ0gmOWM9SiRHTfra4d0YurDIkYaxsSUMW6u+JDCOtp1FgOyM0Y73qzcX/vF5qwms/4zJDZN0uShMBTExmT9PhlwxasTUEqSK21sJHaNCamxEJRuCt/ryOmnXa5b8+7rlcZNHkcRzuAcLsGDK2jAHTShBRQEPMrvDmPzovz7nwsWwtOPnMKf+B8/gBFTI9y</latexit>
slide-25
SLIDE 25

Convergence Rate III

slide-26
SLIDE 26

Experiments

slide-27
SLIDE 27

Poster: Pacific Ballroom #95 (Today 6:30–9:00 PM)

Nonconvex Variance Reduced Optimization with Arbitrary Sampling

Samuel Horváth1 Peter Richtárik1,2,3

1 KAUST 2University of Edinburgh 3Moscow Institute of Physics and Technology

The Problem

min xœRd f(x) := 1 n n X i=1 fi(x) (1)
  • fi is Li-smooth but non-convex
  • n is big

Arbitrary Sampling

  • Sampling: a random set-valued mapping S with
values being subsets of [n] := {1, 2, . . . , n}. A sampling is used to generate minibatches in each iteration.
  • Probability matrix associated with sampling S :
Pij def = Prob({i, j} ™ S)
  • Probability vector associated with sampling S :
p = (p1, . . . , pn), pi def = Prob(i œ S)
  • Minibatch size: b = E [|S|] (expected size of S)
  • Proper sampling: Sampling for which pi > 0 for
all i œ [n]
  • “Arbitrary sampling” = any proper sampling

Main Contributions

  • We develop arbitrary sampling variants of 3
popular variance-reduced methods for solving the non-convex problem (1): SVRG [1], SAGA [2], SARAH [3].
  • We are able calculate the optimal sampling out
  • f all samplings of a given minibatch size. This
is the first time an optimal minibatch sampling was computed (from the class of all samplings).
  • We design importance sampling & approximate
importance sampling for minibatches, which vastly outperform standard uniform minibatch strategies in practice.

Key Lemma

Let ’1, ’2, . . . , ’n be vectors in Rd and let ¯ ’ def = 1 n Pn i=1 ’i be their average. Let S be a proper sam-
  • pling. Let v = (v1, . . . , vn) > 0 be such that
P ≠ pp€ ∞ Diag(p1v1, p2v2, . . . , pnvn). (2) Then E 2 6 4
  • X
iœS ’i npi ≠ ¯ ’
  • 23
7 5 Æ 1 n2 n X i=1 vi pi Î’iÎ2. Whenever (2) holds, it must be the case that vi Ø 1 ≠ pi.

Optimal Sampling & Superlinear Speedup

  • Under our analysis, the independent sampling
Sú defined by pi def = 8 > > > < > > > : (b + k ≠ n) Li Pk j=1 Lj, if i Æ k 1, if i > k , is optimal, where k is the largest integer satisfying 0 < b + k ≠ n Æ Pk i=1 Li Lk .
  • All 3 methods enjoy superlinear speed in b up
to the minibatch size bmax := max{b | bLn Æ Pn i=1 Li}.

# Stochastic Gradient Evaluations to Achieve E

ÎÒf(x)Î2

  • Æ ‘
Alg Uniform sampling Arbitrary sampling [NEW] Sú (Best Sampling) [NEW] SVRG max ( n, (1+4/3)Lmaxc1n2/3 ‘ ) [1] max ( n, (1+4–/3)¯ Lc1n2/3 ‘ ) max 8 < :n, (1+4(n≠b) 3n )¯ Lc1n2/3 ‘ 9 = ; SAGA n + 2Lmaxc2n2/3 [2] n + (1+–)¯ Lc2n2/3 ‘ n + (1+n≠b n )¯ Lc2n2/3 ‘ SARAH n + n≠b n≠1L2 maxc3 ‘2 [3] n + –¯ L2c3 ‘2 n + n≠b n ¯ L2c3 ‘2 Constants: Lmax = maxi Li ¯ L = 1 n P i Li c1, c2, c3 = universal constants – := b ¯ L2n2 Pn i=1 viL2 i pi

Samplings

  • Uniform Su: Every subset of [n] of size b
(minibatch size) is chosen with the same probability: 1/(n b)
  • Independent Sú: For each i œ [n] we
independently flip a coin, and with probability pi include element i into S.
  • Approximate Independent Sa: Fix some
k œ [n] and let a = Ák maxiÆk piË. We now sample a single set SÕ of cardinality a using the uniform minibatch sampling Su. Subsequently, we apply an independent sampling Sú to select elements of SÕ, with selection probabilities pÕ i = kpi/a. The resulting random set is Sa.

SVRG with Arbitrary Sampling

Algorithm 1: SVRG ˜ x0 = x0 m = x0, M = ÁT/mË; for s = 0 to M ≠ 1 do xs+1 = xs m; gs+1 = 1 n Pn i=1 Òfi(˜ xs) for t = 0 to m ≠ 1 do Draw a random subset (minibatch) St ≥ S vs+1 t = P itœSt 1 npit (Òfit(xs+1 t ) ≠ Òfit(˜ xs)) + gs+1 xs+1 t+1 = xs+1 t ≠ ÷vs+1 t end ˜ xs+1 = xs+1 m end Output: Iterate xa chosen uniformly random from {{xs+1 t }m t=0}M s=0

Numerical Results References

[1] Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. In The 33th International Conference on Machine Learning, pages 314–323, 2016. [2] Sashank J Reddi, Suvrit Sra, Barnabás Póczos, and Alex Smola. Fast incremental method for smooth nonconvex optimization. In Decision and Control (CDC), 2016 IEEE 55th Conference
  • n, pages 1971–1977. IEEE, 2016.
[3] Lam M Nguyen, Jie Liu, Katya Scheinberg, and Martin Taká. Stochastic recursive gradient algorithm for nonconvex optimization. arXiv:1705.07261, 2017.
slide-28
SLIDE 28

Thank you!