Nonconvex Variance Reduced Optimization with Arbitrary Sampling
Samuel Horváth Peter Richtárik
Nonconvex Variance Reduced Optimization with Arbitrary Sampling - - PowerPoint PPT Presentation
Nonconvex Variance Reduced Optimization with Arbitrary Sampling Samuel Horvth Peter Richtrik Empirical Risk Minimization n x R d f ( x ) := 1 X min f i ( x ) n i =1 Empirical Risk Minimization n x R d f ( x ) := 1 X min f i (
Samuel Horváth Peter Richtárik
n is big
n is big
krfi(x) rfi(y)k Likx yk
non-convex, Li-smooth
SVRG SAGA SARAH
Nguyen, Liu, Scheinberg & Takáč ICML 2017 Defazio, Bach & Lacoste-Julien NIPS 2014 Johnson & Zhang NIPS 2013
SVRG SAGA SARAH
Nguyen, Liu, Scheinberg & Takáč ICML 2017 Defazio, Bach & Lacoste-Julien NIPS 2014 Johnson & Zhang NIPS 2013
Uniform sampling Uniform sampling Uniform sampling
SVRG SAGA SARAH
Nguyen, Liu, Scheinberg & Takáč 2017 Reddi, Hefny, Sra, Poczos, Smola CDC 2016 Konečný & Richtárik FAMS 2017
SVRG SAGA SARAH
Nguyen, Liu, Scheinberg & Takáč 2017 Reddi, Hefny, Sra, Poczos, Smola CDC 2016 Konečný & Richtárik FAMS 2017
Uniform sampling Uniform sampling Uniform sampling Mini-batch size
Richtárik & Takáč (OL 2016; arXiv 2013) Qu, Richtárik & Zhang (NIPS 2015) Qu & Richtárik (COAP 2016) Chambolle, Ehrhardt, Richtárik & Schoenlieb (SIOPT 2018) Hanzely & Richtárik (AISTATS 2019) Qian, Qu & Richtárik (ICML 2019) Gower, Loizou, Qian, Sailanbayev, Shulgin & Richtárik (ICML 2019)
Richtárik & Takáč (OL 2016; arXiv 2013) Qu, Richtárik & Zhang (NIPS 2015) Qu & Richtárik (COAP 2016) Chambolle, Ehrhardt, Richtárik & Schoenlieb (SIOPT 2018) Hanzely & Richtárik (AISTATS 2019) Qian, Qu & Richtárik (ICML 2019) Gower, Loizou, Qian, Sailanbayev, Shulgin & Richtárik (ICML 2019)
First optimal/importance sampling for minibatches!
Sampling: a random subset of {1, 2, … , n}
Pij := Prob({i, j} ⊆ S)
pi := Prob({i} ⊆ S) = Pii
Probability matrix associated with sampling Probability vector associated with sampling P ∈ Rn×n
<latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit>S
<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>p ∈ Rn
<latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit>Proper sampling: pi > 0
<latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit>for all i = 1, 2, . . . , n
<latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit>S
<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>Sampling: a random subset of {1, 2, … , n}
A sampling is uniquely defined by assigning probabilities to all 2n subsets of {1, 2, ... , n}
Pij := Prob({i, j} ⊆ S)
pi := Prob({i} ⊆ S) = Pii
Probability matrix associated with sampling Probability vector associated with sampling P ∈ Rn×n
<latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit>S
<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>p ∈ Rn
<latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit>Proper sampling: pi > 0
<latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit>for all i = 1, 2, . . . , n
<latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit>S
<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>Sampling: a random subset of {1, 2, … , n}
A sampling is uniquely defined by assigning probabilities to all 2n subsets of {1, 2, ... , n}
Pij := Prob({i, j} ⊆ S)
pi := Prob({i} ⊆ S) = Pii
Probability matrix associated with sampling Probability vector associated with sampling P ∈ Rn×n
<latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit>S
<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>p ∈ Rn
<latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit>Proper sampling: pi > 0
<latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit>for all i = 1, 2, . . . , n
<latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit>S
<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>Standard sampling:
Examples
S = {i} with probability 1
n for all i = 1, 2, . . . , n
<latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit><latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit><latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit><latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit>S = C with probability
1
(
n b)for all C ⊂ {1, 2, . . . , n} such that |C| = b
<latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit><latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit><latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit><latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit>Standard mini-batch sampling:
Sampling: a random subset of {1, 2, … , n}
A sampling is uniquely defined by assigning probabilities to all 2n subsets of {1, 2, ... , n}
Pij := Prob({i, j} ⊆ S)
pi := Prob({i} ⊆ S) = Pii
Probability matrix associated with sampling Probability vector associated with sampling P ∈ Rn×n
<latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit><latexit sha1_base64="Yr07Ewa2bp/z+RzVCN36bBAh87w=">ACDHicbVC7TsNAEFzDOEVoKQ5kSBRXYaKCNoKAMiDyk20flyTk45n627M1Jk+QNo+BUaChCi5QPo+BvOiQtIGml0cyudnf8mDOlbfvbWldW9/YLG2Vt3d29/YrB4cdFSWS0DaJeCR7PlaUM0HbmlOe7GkOPQ57fqTq9zvPlCpWCTu9DSmXohHgWMYG2kQaVaS10/QK3MZQK5IdZj309vs/tUuJqFVCGR1UyXbdnQMvEKUgVCrQGlS93GJEkpEITjpXqO3asvRLzQinWdlNFI0xmeAR7RsqsFnkpbNnMnRqlCEKImlKaDRTf0+kOFRqGvqmMz9XLXq5+J/XT3Rw4aVMxImgswXBQlHOkJ5MmjIJCWaTw3BRDJzKyJjLDHRJr+yCcFZfHmZdBp1x647N41q87KIowTHcAJn4MA5NOEaWtAGAo/wDK/wZj1ZL9a79TFvXbGKmSP4A+vzBzPMmxI=</latexit>S
<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>p ∈ Rn
<latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit><latexit sha1_base64="0/qQEvbX90F5JmZDCWI2z8AWBbc=">AB/HicbVC7TsMwFL3hWcor0JHFokViqpIuMFawMBZEH1ITKsd1WquOE9kOUhWVX2FhACFWPoSNv8FpM0DLkSwdnXOv7vEJEs6Udpxva219Y3Nru7RT3t3bPzi0j47Kk4loW0S81j2AqwoZ4K2NdOc9hJcRw2g0m17nfaRSsVjc62lC/QiPBAsZwdpIA7tSzwmkBdhPQ6C7G72IGoDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW9lJFE0wmeET7hgocUeVn8/AzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlIte7n4n9dPdXjpZ0wkqaCLA6FKUc6RnkTaMgkJZpPDcFEMpMVkTGWmGjTV9mU4C5/eZV0GnXqbu3jWrzqijBCdwCufgwgU04QZa0AYCU3iGV3iznqwX6936WIyuWcVOBf7A+vwB+v2Ug=</latexit>Proper sampling: pi > 0
<latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit><latexit sha1_base64="beBScLQFAC6k7ZcYSqCYMD7rOIA=">ACAnicbVC7SgNBFJ31GdfXqpXYDCYBixB202ijBG0sI5gHJEuYncwmQ2Znl5m7QgjBxl+xsVDE1q+w82+cJFto4oGBwzn3cOeIBFcg+t+Wyura+sbm7kte3tnd2/fOThs6DhVlNVpLGLVCohmgktWBw6CtRLFSBQI1gyGN1O/+cCU5rG8h1HC/Ij0JQ85JWCkrnNcSLr8yi1gu1jgl16pUur0YtAlWeg6ebfszoCXiZeRPMpQ6zpfJkrTiEmgmjd9twE/DFRwKlgE7uTapYQOiR91jZUkohpfzw7YKLRunhMFbmScAz9XdiTCKtR1FgJiMCA73oTcX/vHYK4YU/5jJgUk6XxSmAkOMp3gHleMghgZQqji5q+YDogiFExrtinBWzx5mTQqZc8te3eVfPU6qyOHTtApOkMeOkdVdItqI4oekTP6BW9WU/Wi/VufcxHV6wsc4T+wPr8AWZ8lDk=</latexit>for all i = 1, 2, . . . , n
<latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit><latexit sha1_base64="K2WQjwNQGLGsN/LsJyrm5k20Us=">AB+HicbVDLSsNAFL2pr1ofjbp0E2wF6Uk3ehGKLpxWcE+oA1lMp20QyeTMHMj1NIvceNCEbd+ijv/xmbhbYeGDicw/3zgkSwTW67reV29jc2t7J7xb29g8Oi/bRcUvHqaKsSWMRq05ANBNcsiZyFKyTKEaiQLB2ML6d+1HpjSP5QNOEuZHZCh5yClBI/XtYplfe5VapTeIUVdkuW+X3Kq7gLNOvIyUIEOjb3+ZKE0jJpEKonXcxP0p0Qhp4LNCr1Us4TQMRmyrqGSREz708XhM+fcKAMnjJV5Ep2F+jsxJZHWkygwkxHBkV715uJ/XjfF8MqfcpmkyCRdLgpT4WDszFtwBlwximJiCKGKm1sdOiKUDRdFUwJ3uqX10mrVvXcqndfK9VvsjrycApncAEeXEId7qABTaCQwjO8wpv1ZL1Y79bHcjRnZkT+APr8wdgG5GY</latexit>S
<latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit><latexit sha1_base64="3BcuTsWrhY3Ba7ZEjiDUCBKuXew=">AB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYk2lhgESeBC9pY52LC3d9ndMyEXfoKNhcbY+ovs/DcucIWCL5nk5b2ZzMwLEsG1cd1vp7CxubW9U9wt7e0fHB6Vj086Ok4VwzaLRay6AdUouMS24UZgN1FIo0DgYzC5nfuPT6g0j+WDmSboR3QkecgZNVZqVvVQbni1twFyDrxclKBHM1B+as/jFkaoTRMUK17npsYP6PKcCZwVuqnGhPKJnSEPUsljVD72eLUGbmwypCEsbIlDVmovycyGmk9jQLbGVEz1qveXPzP6UmvPYzLpPUoGTLRWEqiInJ/G8y5AqZEVNLKFPc3krYmCrKjE2nZEPwVl9eJ516zXNr3n290rjJ4yjCGZzDJXhwBQ24gya0gcEInuEV3hzhvDjvzseyteDkM6fwB87nD2UZjTE=</latexit>Arbitrary sampling paradigm = perform iteration complexity analysis for any proper sampling
Standard sampling:
Examples
S = {i} with probability 1
n for all i = 1, 2, . . . , n
<latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit><latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit><latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit><latexit sha1_base64="wYRUx/pK9Vh5A+mivJphDXvU0v4=">ACK3icbVC7TsMwFHV4lvIqMLJYNEgMVZV0gaVSVRZGEBQqNVHluE5r4diRfQOKov4PC7/CAMPsfIfuKUDrzMdnXOP7r0nSgU34Hmvztz8wuLScmlvLq2vrFZ2dq+NCrTlHWoEkp3I2KY4J1gINg3VQzkSCXUXxP/6oZpw5W8gDxlYUKGksecErBSv9J2z3ETBwUPxi6+5TDCqVYRibjgkGM3iDWhT8upLVjpTERAru86dcatWCgwNSk269Uvbo3Bf5L/BmpohlO+5VHG6VZwiRQYzp+V4KYUE0cCrYuBxkhqWEXpMh61kqScJMWEx/HeN9qwymp8RKAp6q3xMFSYzJk8hOJgRG5rc3Ef/zehnER2HBZoBk/RrUZwJDApPisMDrhkFkVtCqOb2VkxHxNYDt6yLcH/fJfctmo+17dP2tUW+1ZHSW0i/bQAfLRIWqhE3SKOoiO/SAntGLc+8OW/O+9fonDPL7KAfcD4+ATh6pTM=</latexit>S = C with probability
1
(
n b)for all C ⊂ {1, 2, . . . , n} such that |C| = b
<latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit><latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit><latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit><latexit sha1_base64="AyQ2rK4zKG5Ut1dM9MnPI4d+xRk=">ACV3icbVFBT9swGHUCY13YRtmOXD7RTNqhqpJe4IKE6GVHplFAaqrKdh1i4diR/WoCvmTiAt/ZfNKT1sHd6eu9udnVinpMEmegnBr+83O2967aPf9h497/f1Pl87UlospN8rYa0adUFKLKUpU4rqygpZMiSt2O+n8q5/COmn0Ba4qMS/pjZa5BS9tOjr+AecwCSGO4kFVNYwyqSuI4y3lTdo2jYaMF8Y4AaxtY4Aoyo0FqhTEk8zVzAmErEmH42G2NOiGoDMfiyJX8wKwoAjx/eT+hMXRoj9IRska8JqkGzIgG5wv+g/+SF6XQiNX1LlZmlQ4b6hFyZVo6x2oqL8lt6ImaealsLNm3UvLXzxyhK6XOjEdbq3xMNLZ1blcwnS4qFe+l14v+8WY358byRuqpRaP58UV4rQANdybCUVnBUK08ot9LvCrygvk70X9GVkL58mtyOR6lySj9Ph6cnm3q6JEDcki+kpQckVPyjZyTKeHkfwKtoLt4Cn4He6EvedoGxmPpN/EO7/ATAXr0g=</latexit>Standard mini-batch sampling:
SVRG with Arbitrary Sampling x+ = x η X
i∈S
1 npi (rfi(x) rfi(ˆ x)) + rf(ˆ x) !
<latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit>Unbiased estimator of the gradient
SVRG with Arbitrary Sampling x+ = x η X
i∈S
1 npi (rfi(x) rfi(ˆ x)) + rf(ˆ x) !
<latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit>Unbiased estimator of the gradient
SVRG with Arbitrary Sampling x+ = x η X
i∈S
1 npi (rfi(x) rfi(ˆ x)) + rf(ˆ x) !
<latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit><latexit sha1_base64="forS/lGdcRLu94d3yhmsHgoAnk=">ACh3icbVFNbxMxEPUuBdrwlcKRy4gIKaEi7JYKekEqcOFYBGkrxcvK63gTq17vyp5FiSz/FX4UN/4NTrqU0jKSpef35nk8M0WjpMUk+RXFt7Zu37m7vdO7d/Bw0f93cntm4NFxNeq9qcFcwKJbWYoEQlzhojWFUocVqcf1zrp9+FsbLWX3HViKxicy1LyRkGKu/oNPltz14B0t4CVQgA6BKlDiktq1yJ4FKDY5uCjkjZh6+eE9Lw7hLvbsq6CaX3v9xa1YoBmUuh8vR+uW/d7pg6JZ+BNTI+QJHsHepXmqdRLO8P0jGySbgJkg7MCBdHOf9n3RW87YSGrli1k7TpMHMYOSK+F7tLWiYfyczcU0QM0qYTO36cLD8DMoKxNOBphw151OFZu6qKkFkxXNjr2pr8nzZtsTzMnNRNi0Lzi0JlqwBrWC8FZtIjmoVAONGhr8CX7AwZAyr64UhpNdbvglO9sdpMk4/7w+OPnTj2CZPyTMyJCl5S47IJ3JMJoRHW9GL6HV0EO/Er+I38eFahx1nifkn4jf/wZTysJS</latexit>Arbitrary sampling
pi = 1
n for all i
<latexit sha1_base64="LmnDBt30CO6XQ28Rzf/6r/4U7Hs=">ACHicbVC7TsMwFHV4lvIKMDJg0SAxVUkXWJAqWBiLRB9SG0WO67RWHTuyHaQqysjCr7AwgBArn8DG3+CmGaDlTEfn3Kt7zwkTRpV23W9rZXVtfWOzslXd3tnd27cPDjtKpBKTNhZMyF6IFGUk7ampFeIgmKQ0a64eRm5ncfiFRU8Hs9TYgfoxGnEcVIGymwT5wkoPAKDiKJcOblGc8dGAkJEWPQoU5g19y6WwAuE68kNVCiFdhfg6HAaUy4xgwp1fcRPsZkpiRvLqIFUkQXiCRqRvKEcxUX5WBMnhmVGxflIcA0L9fdGhmKlpnFoJmOkx2rRm4n/ef1UR5d+RnmSasLx/FCUMqgFnLUCh1QSrNnUEIQlNb9CPEamEm26q5oSvMXIy6TqHtu3btr1JrXZR0VcAxOwTnwAVoglvQAm2AwSN4Bq/gzXqyXqx362M+umKVO0fgD6zPH+Iul/Q=</latexit><latexit sha1_base64="LmnDBt30CO6XQ28Rzf/6r/4U7Hs=">ACHicbVC7TsMwFHV4lvIKMDJg0SAxVUkXWJAqWBiLRB9SG0WO67RWHTuyHaQqysjCr7AwgBArn8DG3+CmGaDlTEfn3Kt7zwkTRpV23W9rZXVtfWOzslXd3tnd27cPDjtKpBKTNhZMyF6IFGUk7ampFeIgmKQ0a64eRm5ncfiFRU8Hs9TYgfoxGnEcVIGymwT5wkoPAKDiKJcOblGc8dGAkJEWPQoU5g19y6WwAuE68kNVCiFdhfg6HAaUy4xgwp1fcRPsZkpiRvLqIFUkQXiCRqRvKEcxUX5WBMnhmVGxflIcA0L9fdGhmKlpnFoJmOkx2rRm4n/ef1UR5d+RnmSasLx/FCUMqgFnLUCh1QSrNnUEIQlNb9CPEamEm26q5oSvMXIy6TqHtu3btr1JrXZR0VcAxOwTnwAVoglvQAm2AwSN4Bq/gzXqyXqx362M+umKVO0fgD6zPH+Iul/Q=</latexit><latexit sha1_base64="LmnDBt30CO6XQ28Rzf/6r/4U7Hs=">ACHicbVC7TsMwFHV4lvIKMDJg0SAxVUkXWJAqWBiLRB9SG0WO67RWHTuyHaQqysjCr7AwgBArn8DG3+CmGaDlTEfn3Kt7zwkTRpV23W9rZXVtfWOzslXd3tnd27cPDjtKpBKTNhZMyF6IFGUk7ampFeIgmKQ0a64eRm5ncfiFRU8Hs9TYgfoxGnEcVIGymwT5wkoPAKDiKJcOblGc8dGAkJEWPQoU5g19y6WwAuE68kNVCiFdhfg6HAaUy4xgwp1fcRPsZkpiRvLqIFUkQXiCRqRvKEcxUX5WBMnhmVGxflIcA0L9fdGhmKlpnFoJmOkx2rRm4n/ef1UR5d+RnmSasLx/FCUMqgFnLUCh1QSrNnUEIQlNb9CPEamEm26q5oSvMXIy6TqHtu3btr1JrXZR0VcAxOwTnwAVoglvQAm2AwSN4Bq/gzXqyXqx362M+umKVO0fgD6zPH+Iul/Q=</latexit><latexit sha1_base64="LmnDBt30CO6XQ28Rzf/6r/4U7Hs=">ACHicbVC7TsMwFHV4lvIKMDJg0SAxVUkXWJAqWBiLRB9SG0WO67RWHTuyHaQqysjCr7AwgBArn8DG3+CmGaDlTEfn3Kt7zwkTRpV23W9rZXVtfWOzslXd3tnd27cPDjtKpBKTNhZMyF6IFGUk7ampFeIgmKQ0a64eRm5ncfiFRU8Hs9TYgfoxGnEcVIGymwT5wkoPAKDiKJcOblGc8dGAkJEWPQoU5g19y6WwAuE68kNVCiFdhfg6HAaUy4xgwp1fcRPsZkpiRvLqIFUkQXiCRqRvKEcxUX5WBMnhmVGxflIcA0L9fdGhmKlpnFoJmOkx2rRm4n/ef1UR5d+RnmSasLx/FCUMqgFnLUCh1QSrNnUEIQlNb9CPEamEm26q5oSvMXIy6TqHtu3btr1JrXZR0VcAxOwTnwAVoglvQAm2AwSN4Bq/gzXqyXqx362M+umKVO0fgD6zPH+Iul/Q=</latexit>Standard sampling:
# data points
Ekrf(x)k2 ✏
<latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit><latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit><latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit><latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit># data points
Ekrf(x)k2 ✏
<latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit><latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit><latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit><latexit sha1_base64="Ti37wOSTf5wqbv62wQeJIRgk/M=">ACEnicbVC7TgJBFJ3F+ILtbSZCbQkF0aLYnGxBITeSQsktnhLkyYnV1nZo1k4Rts/BUbC42xtbLzbxwehYInucnJOfm3nu8iDOlbfvbSq2srq1vpDczW9s7u3vZ/YO6CmNJoUZDHsqmRxRwJqCmebQjCSQwOPQ8AYXE79xD1KxUNzoYQTtgPQE8xkl2kidbDGfuDLAl2PsjlxBPE6wX3gouqPbMnY53GEXIsV4KPKdbM4u2VPgZeLMSQ7NUe1kv9xuSOMAhKacKNVy7Ei3EyI1oxzGTdWEBE6ID1oGSpIAKqdTF8a4xOjdLEfSlNC46n6eyIhgVLDwDOdAdF9tehNxP+8Vqz9s3bCRBRrEHS2yI851iGe5IO7TALVfGgIoZKZWzHtE0moNilmTAjO4svLpF4uOXbJuS7nKufzONLoCB2jAnLQKaqgK1RFNUTRI3pGr+jNerJerHfrY9asuYzh+gPrM8f8D2cYA=</latexit>KEY QUANTITY: DEPENDS ON THE SAMPLING
n
i=1
i
n
i=1
i
Pij := Prob({i, j} ⊆ S) pi := Prob({i} ⊆ S) = Pii P pp> Diag(p1v1, p2v2, . . . , pnvn).
Constants satisfying: Expected mini-batch size:
b = E|S|
<latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit><latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit><latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit><latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit># data points
¯ L = 1 n
n
X
i=1
Li
<latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit><latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit><latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit><latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit>krfi(x) rfi(y)k Likx yk
n
i=1
i
Optimal rate: minimize over {(vi, pi)}n
i=1
Pij := Prob({i, j} ⊆ S) pi := Prob({i} ⊆ S) = Pii P pp> Diag(p1v1, p2v2, . . . , pnvn).
Constants satisfying: Expected mini-batch size:
b = E|S|
<latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit><latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit><latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit><latexit sha1_base64="tpgusjk+WuyjR/GurkQVcVD3bUs=">AB+XicbVBNTwIxEJ3FL8SvVY9eGsHE9nlohcTojHxiFGQBDakWwo0tN1N2yUhC/EiweN8eo/8ea/scAeFHzJC/vzWRmXhzpo3nfTu5tfWNza38dmFnd2/wD08augoUYTWScQj1QyxpxJWjfMcNqMFcUi5PQpHN7M/KcRVZpF8tGMYxoI3Jesxwg2Vuq4bim8QmlbCXQ7RZOHSanjFr2yNwdaJX5GipCh1nG/2t2IJIJKQzjWuV7sQlSrAwjnE4L7UTGJMh7tOWpRILqoN0fvkUnVmli3qRsiUNmqu/J1IstB6L0HYKbAZ62ZuJ/3mtxPQug5TJODFUksWiXsKRidAsBtRlihLDx5Zgopi9FZEBVpgYG1bBhuAv7xKGpWy75X9+0qxep3FkYcTOIVz8OECqnAHNagDgRE8wyu8Oanz4rw7H4vWnJPNHMfOJ8/mdqSUw=</latexit># data points
¯ L = 1 n
n
X
i=1
Li
<latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit><latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit><latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit><latexit sha1_base64="Lz5G7p8kgFaAGFRymOKOJEsgpI=">ACEHicbVC7TsMwFHV4lvIKMLJYVAimKu4CS6UKFoYORaIPqQmR4zqtVceJbAepivIJLPwKCwMIsTKy8Te4bQZoOdKVjs65V/feEyScKe0439bK6tr6xmZpq7y9s7u3bx8cdlScSkLbJOax7AVYUc4EbWumOe0lkuIo4LQbjK+nfveBSsVicacnCfUiPBQsZARrI/n2mdt3AyzZl53Q4lJhvJM5K5KIz9jdZTfC9j0mev5dsWpOjPAZYIKUgEFWr795Q5ikZUaMKxUn3kJNrLsNSMcJqX3VTRBJMxHtK+oQJHVHnZ7KEcnhplAMNYmhIaztTfExmOlJpEgemMsB6pRW8q/uf1Ux1ehkTSaqpIPNFYcqhjuE0HThgkhLNJ4ZgIpm5FZIRNrlok2HZhIAWX14mnVoVOV0W6s0ro4SuAYnIBzgMAFaIAb0AJtQMAjeAav4M16sl6sd+tj3rpiFTNH4A+szx/jkZ0X</latexit>krfi(x) rfi(y)k Likx yk
Poster: Pacific Ballroom #95 (Today 6:30–9:00 PM)
Nonconvex Variance Reduced Optimization with Arbitrary Sampling
Samuel Horváth1 Peter Richtárik1,2,3
1 KAUST 2University of Edinburgh 3Moscow Institute of Physics and TechnologyThe Problem
min xœRd f(x) := 1 n n X i=1 fi(x) (1)Arbitrary Sampling
Main Contributions
Key Lemma
Let ’1, ’2, . . . , ’n be vectors in Rd and let ¯ ’ def = 1 n Pn i=1 ’i be their average. Let S be a proper sam-Optimal Sampling & Superlinear Speedup
# Stochastic Gradient Evaluations to Achieve E
ÎÒf(x)Î2
Samplings
SVRG with Arbitrary Sampling
Algorithm 1: SVRG ˜ x0 = x0 m = x0, M = ÁT/mË; for s = 0 to M ≠ 1 do xs+1 = xs m; gs+1 = 1 n Pn i=1 Òfi(˜ xs) for t = 0 to m ≠ 1 do Draw a random subset (minibatch) St ≥ S vs+1 t = P itœSt 1 npit (Òfit(xs+1 t ) ≠ Òfit(˜ xs)) + gs+1 xs+1 t+1 = xs+1 t ≠ ÷vs+1 t end ˜ xs+1 = xs+1 m end Output: Iterate xa chosen uniformly random from {{xs+1 t }m t=0}M s=0Numerical Results References
[1] Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. In The 33th International Conference on Machine Learning, pages 314–323, 2016. [2] Sashank J Reddi, Suvrit Sra, Barnabás Póczos, and Alex Smola. Fast incremental method for smooth nonconvex optimization. In Decision and Control (CDC), 2016 IEEE 55th Conference