Neural Networks as Stat Mech Systems Based on arXiv:1710.06570 - - PowerPoint PPT Presentation

▶

Nov 28, 2022 277 likes •463 views

Neural Networks as Stat Mech Systems Based on arXiv:1710.06570 [stat.ML], A Correspondence Between Random Neural Networks and Statistical Field Theory Yoni Kahn, KICP/UIUC hep-ai 1/8/19 <latexit

SLIDE 1

Neural Networks  as  Stat Mech Systems

Based on arXiv:1710.06570 [stat.ML], “A Correspondence Between Random Neural Networks and Statistical Field Theory”

Yoni Kahn, KICP/UIUC  hep-ai 1/8/19

SLIDE 2

Motivation

Neural networks have extremely large numbers 

f parameters. On the surface, that makes them hard to

understand analytically. But physics provides examples where systems simplify  in the limit of large numbers of constituents:   thermodynamics/stat mech. N ∼ 1023

<latexit sha1_base64="vB/vSKiYeEnOclwCJEXaOeJo0=">AB9XicdVDLSgMxFM34rPVdekmWARXZWY6tHVXdONKtgHtNOSTNtaJIZkoxShv6HGxeKuPVf3Pk3ZtoKnrgwuGce7n3niBmVGnb/rBWVtfWNzZzW/ntnd29/cLBYUtFicSkiSMWyU6AFGFUkKampFOLAniASPtYHKZ+e07IhWNxK2exsTnaCRoSDHSRupfw56iHDp2P3XLs0GhaJfOaxXq0C7ZNtVx3Uy4la9sgcdo2QogiUag8J7bxjhBOhMUNKdR071n6KpKaYkVm+lygSIzxBI9I1VCBOlJ/Or57BU6MYRhJU0LDufp9IkVcqSkPTCdHeqx+e5n4l9dNdFjzUyriRBOBF4vChEdwSwCOKSYM2mhiAsqbkV4jGSCGsTVN6E8PUp/J+03Jj+I1XrF8s48iBY3ACzoADqAOrkADNAEGEjyAJ/Bs3VuP1ov1umhdsZYzR+AHrLdPFSRlw=</latexit><latexit sha1_base64="vB/vSKiYeEnOclwCJEXaOeJo0=">AB9XicdVDLSgMxFM34rPVdekmWARXZWY6tHVXdONKtgHtNOSTNtaJIZkoxShv6HGxeKuPVf3Pk3ZtoKnrgwuGce7n3niBmVGnb/rBWVtfWNzZzW/ntnd29/cLBYUtFicSkiSMWyU6AFGFUkKampFOLAniASPtYHKZ+e07IhWNxK2exsTnaCRoSDHSRupfw56iHDp2P3XLs0GhaJfOaxXq0C7ZNtVx3Uy4la9sgcdo2QogiUag8J7bxjhBOhMUNKdR071n6KpKaYkVm+lygSIzxBI9I1VCBOlJ/Or57BU6MYRhJU0LDufp9IkVcqSkPTCdHeqx+e5n4l9dNdFjzUyriRBOBF4vChEdwSwCOKSYM2mhiAsqbkV4jGSCGsTVN6E8PUp/J+03Jj+I1XrF8s48iBY3ACzoADqAOrkADNAEGEjyAJ/Bs3VuP1ov1umhdsZYzR+AHrLdPFSRlw=</latexit><latexit sha1_base64="vB/vSKiYeEnOclwCJEXaOeJo0=">AB9XicdVDLSgMxFM34rPVdekmWARXZWY6tHVXdONKtgHtNOSTNtaJIZkoxShv6HGxeKuPVf3Pk3ZtoKnrgwuGce7n3niBmVGnb/rBWVtfWNzZzW/ntnd29/cLBYUtFicSkiSMWyU6AFGFUkKampFOLAniASPtYHKZ+e07IhWNxK2exsTnaCRoSDHSRupfw56iHDp2P3XLs0GhaJfOaxXq0C7ZNtVx3Uy4la9sgcdo2QogiUag8J7bxjhBOhMUNKdR071n6KpKaYkVm+lygSIzxBI9I1VCBOlJ/Or57BU6MYRhJU0LDufp9IkVcqSkPTCdHeqx+e5n4l9dNdFjzUyriRBOBF4vChEdwSwCOKSYM2mhiAsqbkV4jGSCGsTVN6E8PUp/J+03Jj+I1XrF8s48iBY3ACzoADqAOrkADNAEGEjyAJ/Bs3VuP1ov1umhdsZYzR+AHrLdPFSRlw=</latexit><latexit sha1_base64="vB/vSKiYeEnOclwCJEXaOeJo0=">AB9XicdVDLSgMxFM34rPVdekmWARXZWY6tHVXdONKtgHtNOSTNtaJIZkoxShv6HGxeKuPVf3Pk3ZtoKnrgwuGce7n3niBmVGnb/rBWVtfWNzZzW/ntnd29/cLBYUtFicSkiSMWyU6AFGFUkKampFOLAniASPtYHKZ+e07IhWNxK2exsTnaCRoSDHSRupfw56iHDp2P3XLs0GhaJfOaxXq0C7ZNtVx3Uy4la9sgcdo2QogiUag8J7bxjhBOhMUNKdR071n6KpKaYkVm+lygSIzxBI9I1VCBOlJ/Or57BU6MYRhJU0LDufp9IkVcqSkPTCdHeqx+e5n4l9dNdFjzUyriRBOBF4vChEdwSwCOKSYM2mhiAsqbkV4jGSCGsTVN6E8PUp/J+03Jj+I1XrF8s48iBY3ACzoADqAOrkADNAEGEjyAJ/Bs3VuP1ov1umhdsZYzR+AHrLdPFSRlw=</latexit>

Ψ(x1, x2, . . . , xN)

<latexit sha1_base64="3kXWQu8EeKhJ/SiEfdC0KYMJI=">ACA3icdVDLSgMxFM3UV62vUXe6CRahQikz49DWXdGNK6lgH9AOQyZN29DMgyQjLUPBjb/ixoUibv0Jd/6NmbaCih4IHM65h5t7vIhRIQ3jQ8sLa+srmXcxubW9s7+u5eU4Qx6SBQxbytocEYTQgDUklI+2IE+R7jLS80UXqt24JFzQMbuQkIo6PBgHtU4yklz9oFsXtDB2zSIcu1YRdnuhFCm/OnH1vFE6q5YtuwyNkmFUTMtMiVWxT21oKiVFHixQd/V3lcaxTwKJGRKiYxqRdBLEJcWMTHPdWJAI4REakI6iAfKJcJLZDVN4rJQe7IdcvUDCmfo9kSBfiInvqUkfyaH47aXiX14nlv2qk9AgiUJ8HxRP2ZQhjAtBPYoJ1iyiSIc6r+CvEQcYSlqi2nSvi6FP5PmlbJVPzaztfOF3VkwSE4AgVgqogUtQBw2AwR14AE/gWbvXHrUX7XU+mtEWmX3wA9rbJzsTlfo=</latexit><latexit sha1_base64="3kXWQu8EeKhJ/SiEfdC0KYMJI=">ACA3icdVDLSgMxFM3UV62vUXe6CRahQikz49DWXdGNK6lgH9AOQyZN29DMgyQjLUPBjb/ixoUibv0Jd/6NmbaCih4IHM65h5t7vIhRIQ3jQ8sLa+srmXcxubW9s7+u5eU4Qx6SBQxbytocEYTQgDUklI+2IE+R7jLS80UXqt24JFzQMbuQkIo6PBgHtU4yklz9oFsXtDB2zSIcu1YRdnuhFCm/OnH1vFE6q5YtuwyNkmFUTMtMiVWxT21oKiVFHixQd/V3lcaxTwKJGRKiYxqRdBLEJcWMTHPdWJAI4REakI6iAfKJcJLZDVN4rJQe7IdcvUDCmfo9kSBfiInvqUkfyaH47aXiX14nlv2qk9AgiUJ8HxRP2ZQhjAtBPYoJ1iyiSIc6r+CvEQcYSlqi2nSvi6FP5PmlbJVPzaztfOF3VkwSE4AgVgqogUtQBw2AwR14AE/gWbvXHrUX7XU+mtEWmX3wA9rbJzsTlfo=</latexit><latexit sha1_base64="3kXWQu8EeKhJ/SiEfdC0KYMJI=">ACA3icdVDLSgMxFM3UV62vUXe6CRahQikz49DWXdGNK6lgH9AOQyZN29DMgyQjLUPBjb/ixoUibv0Jd/6NmbaCih4IHM65h5t7vIhRIQ3jQ8sLa+srmXcxubW9s7+u5eU4Qx6SBQxbytocEYTQgDUklI+2IE+R7jLS80UXqt24JFzQMbuQkIo6PBgHtU4yklz9oFsXtDB2zSIcu1YRdnuhFCm/OnH1vFE6q5YtuwyNkmFUTMtMiVWxT21oKiVFHixQd/V3lcaxTwKJGRKiYxqRdBLEJcWMTHPdWJAI4REakI6iAfKJcJLZDVN4rJQe7IdcvUDCmfo9kSBfiInvqUkfyaH47aXiX14nlv2qk9AgiUJ8HxRP2ZQhjAtBPYoJ1iyiSIc6r+CvEQcYSlqi2nSvi6FP5PmlbJVPzaztfOF3VkwSE4AgVgqogUtQBw2AwR14AE/gWbvXHrUX7XU+mtEWmX3wA9rbJzsTlfo=</latexit><latexit sha1_base64="3kXWQu8EeKhJ/SiEfdC0KYMJI=">ACA3icdVDLSgMxFM3UV62vUXe6CRahQikz49DWXdGNK6lgH9AOQyZN29DMgyQjLUPBjb/ixoUibv0Jd/6NmbaCih4IHM65h5t7vIhRIQ3jQ8sLa+srmXcxubW9s7+u5eU4Qx6SBQxbytocEYTQgDUklI+2IE+R7jLS80UXqt24JFzQMbuQkIo6PBgHtU4yklz9oFsXtDB2zSIcu1YRdnuhFCm/OnH1vFE6q5YtuwyNkmFUTMtMiVWxT21oKiVFHixQd/V3lcaxTwKJGRKiYxqRdBLEJcWMTHPdWJAI4REakI6iAfKJcJLZDVN4rJQe7IdcvUDCmfo9kSBfiInvqUkfyaH47aXiX14nlv2qk9AgiUJ8HxRP2ZQhjAtBPYoJ1iyiSIc6r+CvEQcYSlqi2nSvi6FP5PmlbJVPzaztfOF3VkwSE4AgVgqogUtQBw2AwR14AE/gWbvXHrUX7XU+mtEWmX3wA9rbJzsTlfo=</latexit>

Z(N, V, T)

<latexit sha1_base64="ah+KiWCtDx9PIzoqxt0DK9fgMP0=">AB73icdVDLSsNAFJ3UV62vqks3g0WoUEoSQ1t3RTeupEJf2IYymU7aoZNJnJkIJfQn3LhQxK2/486/cdJWUNEDFw7n3Mu93gRo1KZ5oeRWVldW9/Ibua2tnd29/L7B20ZxgKTFg5ZKLoekoRTlqKka6kSAo8BjpeJPL1O/cEyFpyJtqGhE3QCNOfYqR0lL3tnhdapeap4N8wSyf1yq2U4Fm2TSrlm2lxK46Zw60tJKiAJZoDPLv/WGI4BwhRmSsmeZkXITJBTFjMxy/ViSCOEJGpGephwFRLrJ/N4ZPNHKEPqh0MUVnKvfJxIUSDkNPN0ZIDWv71U/MvrxcqvuQnlUawIx4tFfsygCmH6PBxSQbBiU0QFlTfCvEYCYSVjinQ/j6FP5P2nbZ0vzGKdQvlnFkwRE4BkVgSqogyvQAC2AQMP4Ak8G3fGo/FivC5aM8Zy5hD8gPH2CZrCjwc=</latexit><latexit sha1_base64="ah+KiWCtDx9PIzoqxt0DK9fgMP0=">AB73icdVDLSsNAFJ3UV62vqks3g0WoUEoSQ1t3RTeupEJf2IYymU7aoZNJnJkIJfQn3LhQxK2/486/cdJWUNEDFw7n3Mu93gRo1KZ5oeRWVldW9/Ibua2tnd29/L7B20ZxgKTFg5ZKLoekoRTlqKka6kSAo8BjpeJPL1O/cEyFpyJtqGhE3QCNOfYqR0lL3tnhdapeap4N8wSyf1yq2U4Fm2TSrlm2lxK46Zw60tJKiAJZoDPLv/WGI4BwhRmSsmeZkXITJBTFjMxy/ViSCOEJGpGephwFRLrJ/N4ZPNHKEPqh0MUVnKvfJxIUSDkNPN0ZIDWv71U/MvrxcqvuQnlUawIx4tFfsygCmH6PBxSQbBiU0QFlTfCvEYCYSVjinQ/j6FP5P2nbZ0vzGKdQvlnFkwRE4BkVgSqogyvQAC2AQMP4Ak8G3fGo/FivC5aM8Zy5hD8gPH2CZrCjwc=</latexit><latexit sha1_base64="ah+KiWCtDx9PIzoqxt0DK9fgMP0=">AB73icdVDLSsNAFJ3UV62vqks3g0WoUEoSQ1t3RTeupEJf2IYymU7aoZNJnJkIJfQn3LhQxK2/486/cdJWUNEDFw7n3Mu93gRo1KZ5oeRWVldW9/Ibua2tnd29/L7B20ZxgKTFg5ZKLoekoRTlqKka6kSAo8BjpeJPL1O/cEyFpyJtqGhE3QCNOfYqR0lL3tnhdapeap4N8wSyf1yq2U4Fm2TSrlm2lxK46Zw60tJKiAJZoDPLv/WGI4BwhRmSsmeZkXITJBTFjMxy/ViSCOEJGpGephwFRLrJ/N4ZPNHKEPqh0MUVnKvfJxIUSDkNPN0ZIDWv71U/MvrxcqvuQnlUawIx4tFfsygCmH6PBxSQbBiU0QFlTfCvEYCYSVjinQ/j6FP5P2nbZ0vzGKdQvlnFkwRE4BkVgSqogyvQAC2AQMP4Ak8G3fGo/FivC5aM8Zy5hD8gPH2CZrCjwc=</latexit><latexit sha1_base64="ah+KiWCtDx9PIzoqxt0DK9fgMP0=">AB73icdVDLSsNAFJ3UV62vqks3g0WoUEoSQ1t3RTeupEJf2IYymU7aoZNJnJkIJfQn3LhQxK2/486/cdJWUNEDFw7n3Mu93gRo1KZ5oeRWVldW9/Ibua2tnd29/L7B20ZxgKTFg5ZKLoekoRTlqKka6kSAo8BjpeJPL1O/cEyFpyJtqGhE3QCNOfYqR0lL3tnhdapeap4N8wSyf1yq2U4Fm2TSrlm2lxK46Zw60tJKiAJZoDPLv/WGI4BwhRmSsmeZkXITJBTFjMxy/ViSCOEJGpGephwFRLrJ/N4ZPNHKEPqh0MUVnKvfJxIUSDkNPN0ZIDWv71U/MvrxcqvuQnlUawIx4tFfsygCmH6PBxSQbBiU0QFlTfCvEYCYSVjinQ/j6FP5P2nbZ0vzGKdQvlnFkwRE4BkVgSqogyvQAC2AQMP4Ak8G3fGo/FivC5aM8Zy5hD8gPH2CZrCjwc=</latexit>

SLIDE 3

What can we do  with a stat mech analogy?

Computing averages/expectation values is “easy.” A stat mech  theory gives us fluctuations, mean-field EFT,  higher-point correlators, 1/N expansion, etc.

Ising model:

H({zl}) = −J 2 X

zlzl+1

<latexit sha1_base64="jK6up03OnlhWi4k3U3wKnPVMajA=">ACaHicjVHLSgMxFM2M7/oaX4i4CRZB8cGkiNaFIrgRVwpWhc5YMmBjMPkoxQ/Af3fkBbvwKM7UFQUvBA7nHtycxPlnEnl+6+OzQ8Mjo2PlGZnJqemfXm5q9lVghCGyTjmbiNsKScpbShmOL0NhcUJxGnN9HDanfPFIhWZeqW5OwR3UhYzgpWlWt7z2UagdALaopOFGp/1/d9hNB2CdDBvm/B4WG9huoGPt1xAwOzCY/gThALTPS50TUTyCJpcfjfGJtikzTfQsa0vOrACQdOHBCVEq2qBfFy3vJWhnpEhoqgjHUjaRn6tQY6EY4dRUgkLSHJMH3KFNC1OcUBnq3mAGrlumDeNM2JMq2GO/dmicSNlNIutMsLqXP7WS/E1rFiquh5qleaFoSj4vigsOVQbLrcM2E5Qo3rUAE8HsrJDcY7tAZf+mYpcweCn8G1zXdpHFl3vVk+P+OsbBKlgDGwCBA3ACzsAFaAC3pxJZ9FZct5dz12Vz6trtPvWQDfyl37AOoasRk=</latexit><latexit sha1_base64="jK6up03OnlhWi4k3U3wKnPVMajA=">ACaHicjVHLSgMxFM2M7/oaX4i4CRZB8cGkiNaFIrgRVwpWhc5YMmBjMPkoxQ/Af3fkBbvwKM7UFQUvBA7nHtycxPlnEnl+6+OzQ8Mjo2PlGZnJqemfXm5q9lVghCGyTjmbiNsKScpbShmOL0NhcUJxGnN9HDanfPFIhWZeqW5OwR3UhYzgpWlWt7z2UagdALaopOFGp/1/d9hNB2CdDBvm/B4WG9huoGPt1xAwOzCY/gThALTPS50TUTyCJpcfjfGJtikzTfQsa0vOrACQdOHBCVEq2qBfFy3vJWhnpEhoqgjHUjaRn6tQY6EY4dRUgkLSHJMH3KFNC1OcUBnq3mAGrlumDeNM2JMq2GO/dmicSNlNIutMsLqXP7WS/E1rFiquh5qleaFoSj4vigsOVQbLrcM2E5Qo3rUAE8HsrJDcY7tAZf+mYpcweCn8G1zXdpHFl3vVk+P+OsbBKlgDGwCBA3ACzsAFaAC3pxJZ9FZct5dz12Vz6trtPvWQDfyl37AOoasRk=</latexit><latexit sha1_base64="jK6up03OnlhWi4k3U3wKnPVMajA=">ACaHicjVHLSgMxFM2M7/oaX4i4CRZB8cGkiNaFIrgRVwpWhc5YMmBjMPkoxQ/Af3fkBbvwKM7UFQUvBA7nHtycxPlnEnl+6+OzQ8Mjo2PlGZnJqemfXm5q9lVghCGyTjmbiNsKScpbShmOL0NhcUJxGnN9HDanfPFIhWZeqW5OwR3UhYzgpWlWt7z2UagdALaopOFGp/1/d9hNB2CdDBvm/B4WG9huoGPt1xAwOzCY/gThALTPS50TUTyCJpcfjfGJtikzTfQsa0vOrACQdOHBCVEq2qBfFy3vJWhnpEhoqgjHUjaRn6tQY6EY4dRUgkLSHJMH3KFNC1OcUBnq3mAGrlumDeNM2JMq2GO/dmicSNlNIutMsLqXP7WS/E1rFiquh5qleaFoSj4vigsOVQbLrcM2E5Qo3rUAE8HsrJDcY7tAZf+mYpcweCn8G1zXdpHFl3vVk+P+OsbBKlgDGwCBA3ACzsAFaAC3pxJZ9FZct5dz12Vz6trtPvWQDfyl37AOoasRk=</latexit><latexit sha1_base64="jK6up03OnlhWi4k3U3wKnPVMajA=">ACaHicjVHLSgMxFM2M7/oaX4i4CRZB8cGkiNaFIrgRVwpWhc5YMmBjMPkoxQ/Af3fkBbvwKM7UFQUvBA7nHtycxPlnEnl+6+OzQ8Mjo2PlGZnJqemfXm5q9lVghCGyTjmbiNsKScpbShmOL0NhcUJxGnN9HDanfPFIhWZeqW5OwR3UhYzgpWlWt7z2UagdALaopOFGp/1/d9hNB2CdDBvm/B4WG9huoGPt1xAwOzCY/gThALTPS50TUTyCJpcfjfGJtikzTfQsa0vOrACQdOHBCVEq2qBfFy3vJWhnpEhoqgjHUjaRn6tQY6EY4dRUgkLSHJMH3KFNC1OcUBnq3mAGrlumDeNM2JMq2GO/dmicSNlNIutMsLqXP7WS/E1rFiquh5qleaFoSj4vigsOVQbLrcM2E5Qo3rUAE8HsrJDcY7tAZf+mYpcweCn8G1zXdpHFl3vVk+P+OsbBKlgDGwCBA3ACzsAFaAC3pxJZ9FZct5dz12Vz6trtPvWQDfyl37AOoasRk=</latexit>

zl = ±1

<latexit sha1_base64="Fq+0rG8P9+iZafaj3FMvXcX5dI=">AB8nicdZDLSgMxFIYzXmu9V26CRbBVUmK2HahFNy4rGAvMB1LJs20oZlkSDJCLX0MNy4UcevTuPNtzLQVPSHwMf/n0POWEiuLEIfXhLyura+u5jfzm1vbObmFv2VUqilrUiWU7oTEMEla1puBeskmpE4FKwdji6zvH3HtOFK3thxwoKYDCSPOCXWf79rYDnsJvEPcKRVRCGMYQa4coYc1GrVMq5CnEVORbBQo1d47/YVTWMmLRXEGB+jxAYToi2ngk3z3dSwhNARGTDfoSQxM8FkNvIUHjunDyOl3ZMWztzvHRMSGzOQ1cZEzs0v7PM/CvzUxtVgwmXSWqZpPOPolRAq2C2P+xzagVYweEau5mhXRINKHWXSnvjvC1KfwfWuUSdnx9WqxfLM6RA4fgCJwADCqgDq5AzQBQo8gCfw7Fnv0XvxXuelS96i5wD8kPf2Cf1/kGU=</latexit><latexit sha1_base64="Fq+0rG8P9+iZafaj3FMvXcX5dI=">AB8nicdZDLSgMxFIYzXmu9V26CRbBVUmK2HahFNy4rGAvMB1LJs20oZlkSDJCLX0MNy4UcevTuPNtzLQVPSHwMf/n0POWEiuLEIfXhLyura+u5jfzm1vbObmFv2VUqilrUiWU7oTEMEla1puBeskmpE4FKwdji6zvH3HtOFK3thxwoKYDCSPOCXWf79rYDnsJvEPcKRVRCGMYQa4coYc1GrVMq5CnEVORbBQo1d47/YVTWMmLRXEGB+jxAYToi2ngk3z3dSwhNARGTDfoSQxM8FkNvIUHjunDyOl3ZMWztzvHRMSGzOQ1cZEzs0v7PM/CvzUxtVgwmXSWqZpPOPolRAq2C2P+xzagVYweEau5mhXRINKHWXSnvjvC1KfwfWuUSdnx9WqxfLM6RA4fgCJwADCqgDq5AzQBQo8gCfw7Fnv0XvxXuelS96i5wD8kPf2Cf1/kGU=</latexit><latexit sha1_base64="Fq+0rG8P9+iZafaj3FMvXcX5dI=">AB8nicdZDLSgMxFIYzXmu9V26CRbBVUmK2HahFNy4rGAvMB1LJs20oZlkSDJCLX0MNy4UcevTuPNtzLQVPSHwMf/n0POWEiuLEIfXhLyura+u5jfzm1vbObmFv2VUqilrUiWU7oTEMEla1puBeskmpE4FKwdji6zvH3HtOFK3thxwoKYDCSPOCXWf79rYDnsJvEPcKRVRCGMYQa4coYc1GrVMq5CnEVORbBQo1d47/YVTWMmLRXEGB+jxAYToi2ngk3z3dSwhNARGTDfoSQxM8FkNvIUHjunDyOl3ZMWztzvHRMSGzOQ1cZEzs0v7PM/CvzUxtVgwmXSWqZpPOPolRAq2C2P+xzagVYweEau5mhXRINKHWXSnvjvC1KfwfWuUSdnx9WqxfLM6RA4fgCJwADCqgDq5AzQBQo8gCfw7Fnv0XvxXuelS96i5wD8kPf2Cf1/kGU=</latexit><latexit sha1_base64="Fq+0rG8P9+iZafaj3FMvXcX5dI=">AB8nicdZDLSgMxFIYzXmu9V26CRbBVUmK2HahFNy4rGAvMB1LJs20oZlkSDJCLX0MNy4UcevTuPNtzLQVPSHwMf/n0POWEiuLEIfXhLyura+u5jfzm1vbObmFv2VUqilrUiWU7oTEMEla1puBeskmpE4FKwdji6zvH3HtOFK3thxwoKYDCSPOCXWf79rYDnsJvEPcKRVRCGMYQa4coYc1GrVMq5CnEVORbBQo1d47/YVTWMmLRXEGB+jxAYToi2ngk3z3dSwhNARGTDfoSQxM8FkNvIUHjunDyOl3ZMWztzvHRMSGzOQ1cZEzs0v7PM/CvzUxtVgwmXSWqZpPOPolRAq2C2P+xzagVYweEau5mhXRINKHWXSnvjvC1KfwfWuUSdnx9WqxfLM6RA4fgCJwADCqgDq5AzQBQo8gCfw7Fnv0XvxXuelS96i5wD8kPf2Cf1/kGU=</latexit>

βH = 1 2 Z dx [m u(x)2 + v u(x)4 + K(ru(x))2 + · · · ]

<latexit sha1_base64="YQ0Nub7VaSohmZE2pjEUuCPBeQ=">ACvXiclZFNb9QwEIadAG1ZaLuFI5cRK6RFrVZJiLoFCajEpRKXVuq2leKwOI6zNXWcyHZKV1H+JOLCv8He7oF+HOhIth69M6/GnslqwbUJgj+e/+jxk5XVtae9Z8/XNzb7Wy9OdNUoyia0EpU6y4hmgks2MdwIdlYrRspMsNPs4ovLn14ypXklj828ZmlJZpIXnBJjpWn/N86YIXAHwEXitA27Nqow1wayK8A70BS2gtavGiVqFmWtsHo/d5uFO/uBKMgGIdR6CAax+/irhlevf0WQbcNl878IFvcwTZ8/W/PEuSCQLOa3s6M6Z5ZTSk0/7A1buAuxAuYCWcTjt/8J5RZuSUMF0ToJg9qkLVGU8G6Hm40qwm9IDOWJSkZDptF+/s4I1VcigqZY8d20L919GSUut5mdnKkphzfTvnxPtySWOKvbTlsm4Mk/S6UdEIMBW4VULOFaNGzC0Qqrh9K9BzYpdo7MJ7dgjh7S/fhZNoFo+igf7n5bjWEOv0Gs0RCEao310gA7RBFHvg/fd494P/7PfOHL61LfW3peohvh/wLBI/IiA=</latexit><latexit sha1_base64="YQ0Nub7VaSohmZE2pjEUuCPBeQ=">ACvXiclZFNb9QwEIadAG1ZaLuFI5cRK6RFrVZJiLoFCajEpRKXVuq2leKwOI6zNXWcyHZKV1H+JOLCv8He7oF+HOhIth69M6/GnslqwbUJgj+e/+jxk5XVtae9Z8/XNzb7Wy9OdNUoyia0EpU6y4hmgks2MdwIdlYrRspMsNPs4ovLn14ypXklj828ZmlJZpIXnBJjpWn/N86YIXAHwEXitA27Nqow1wayK8A70BS2gtavGiVqFmWtsHo/d5uFO/uBKMgGIdR6CAax+/irhlevf0WQbcNl878IFvcwTZ8/W/PEuSCQLOa3s6M6Z5ZTSk0/7A1buAuxAuYCWcTjt/8J5RZuSUMF0ToJg9qkLVGU8G6Hm40qwm9IDOWJSkZDptF+/s4I1VcigqZY8d20L919GSUut5mdnKkphzfTvnxPtySWOKvbTlsm4Mk/S6UdEIMBW4VULOFaNGzC0Qqrh9K9BzYpdo7MJ7dgjh7S/fhZNoFo+igf7n5bjWEOv0Gs0RCEao310gA7RBFHvg/fd494P/7PfOHL61LfW3peohvh/wLBI/IiA=</latexit><latexit sha1_base64="YQ0Nub7VaSohmZE2pjEUuCPBeQ=">ACvXiclZFNb9QwEIadAG1ZaLuFI5cRK6RFrVZJiLoFCajEpRKXVuq2leKwOI6zNXWcyHZKV1H+JOLCv8He7oF+HOhIth69M6/GnslqwbUJgj+e/+jxk5XVtae9Z8/XNzb7Wy9OdNUoyia0EpU6y4hmgks2MdwIdlYrRspMsNPs4ovLn14ypXklj828ZmlJZpIXnBJjpWn/N86YIXAHwEXitA27Nqow1wayK8A70BS2gtavGiVqFmWtsHo/d5uFO/uBKMgGIdR6CAax+/irhlevf0WQbcNl878IFvcwTZ8/W/PEuSCQLOa3s6M6Z5ZTSk0/7A1buAuxAuYCWcTjt/8J5RZuSUMF0ToJg9qkLVGU8G6Hm40qwm9IDOWJSkZDptF+/s4I1VcigqZY8d20L919GSUut5mdnKkphzfTvnxPtySWOKvbTlsm4Mk/S6UdEIMBW4VULOFaNGzC0Qqrh9K9BzYpdo7MJ7dgjh7S/fhZNoFo+igf7n5bjWEOv0Gs0RCEao310gA7RBFHvg/fd494P/7PfOHL61LfW3peohvh/wLBI/IiA=</latexit><latexit sha1_base64="YQ0Nub7VaSohmZE2pjEUuCPBeQ=">ACvXiclZFNb9QwEIadAG1ZaLuFI5cRK6RFrVZJiLoFCajEpRKXVuq2leKwOI6zNXWcyHZKV1H+JOLCv8He7oF+HOhIth69M6/GnslqwbUJgj+e/+jxk5XVtae9Z8/XNzb7Wy9OdNUoyia0EpU6y4hmgks2MdwIdlYrRspMsNPs4ovLn14ypXklj828ZmlJZpIXnBJjpWn/N86YIXAHwEXitA27Nqow1wayK8A70BS2gtavGiVqFmWtsHo/d5uFO/uBKMgGIdR6CAax+/irhlevf0WQbcNl878IFvcwTZ8/W/PEuSCQLOa3s6M6Z5ZTSk0/7A1buAuxAuYCWcTjt/8J5RZuSUMF0ToJg9qkLVGU8G6Hm40qwm9IDOWJSkZDptF+/s4I1VcigqZY8d20L919GSUut5mdnKkphzfTvnxPtySWOKvbTlsm4Mk/S6UdEIMBW4VULOFaNGzC0Qqrh9K9BzYpdo7MJ7dgjh7S/fhZNoFo+igf7n5bjWEOv0Gs0RCEao310gA7RBFHvg/fd494P/7PfOHL61LfW3peohvh/wLBI/IiA=</latexit>

average and take   continuum limit

SLIDE 4

Feed-forward network: setting notation

l = 0

<latexit sha1_base64="Olr127mk9kZLbBuOIl3w31Dy+Ks=">AB7HicdVBNS8NAEJ34WetX1aOXxSJ4KkMbT0IRS8eK5i20Iay2W7apZtN2N0IpfQ3ePGgiFd/kDf/jZu2go+GHi8N8PMvDlTGnb/rBWVtfWNzYLW8Xtnd29/dLBYUslmSTUJwlPZCfEinImqK+Z5rSTSorjkN2OL7O/fY9lYol4k5PUhrEeChYxAjWRvI5ukR2v1S2Kxf1qutVkV2x7ZrjOjlxa965hxyj5CjDEs1+6b03SEgWU6EJx0p1HTvVwRLzQins2IvUzTFZIyHtGuowDFVwXR+7AydGmWAokSaEhrN1e8TUxwrNYlD0xljPVK/vVz8y+tmOqoHUybSTFNBFouijCOdoPxzNGCSEs0nhmAimbkVkRGWmGiT9GE8PUp+p+03Ipj+K1Xblwt4yjAMZzAGThQgwbcQBN8IMDgAZ7g2RLWo/VivS5aV6zlzBH8gPX2CdCyjgc=</latexit><latexit sha1_base64="Olr127mk9kZLbBuOIl3w31Dy+Ks=">AB7HicdVBNS8NAEJ34WetX1aOXxSJ4KkMbT0IRS8eK5i20Iay2W7apZtN2N0IpfQ3ePGgiFd/kDf/jZu2go+GHi8N8PMvDlTGnb/rBWVtfWNzYLW8Xtnd29/dLBYUslmSTUJwlPZCfEinImqK+Z5rSTSorjkN2OL7O/fY9lYol4k5PUhrEeChYxAjWRvI5ukR2v1S2Kxf1qutVkV2x7ZrjOjlxa965hxyj5CjDEs1+6b03SEgWU6EJx0p1HTvVwRLzQins2IvUzTFZIyHtGuowDFVwXR+7AydGmWAokSaEhrN1e8TUxwrNYlD0xljPVK/vVz8y+tmOqoHUybSTFNBFouijCOdoPxzNGCSEs0nhmAimbkVkRGWmGiT9GE8PUp+p+03Ipj+K1Xblwt4yjAMZzAGThQgwbcQBN8IMDgAZ7g2RLWo/VivS5aV6zlzBH8gPX2CdCyjgc=</latexit><latexit sha1_base64="Olr127mk9kZLbBuOIl3w31Dy+Ks=">AB7HicdVBNS8NAEJ34WetX1aOXxSJ4KkMbT0IRS8eK5i20Iay2W7apZtN2N0IpfQ3ePGgiFd/kDf/jZu2go+GHi8N8PMvDlTGnb/rBWVtfWNzYLW8Xtnd29/dLBYUslmSTUJwlPZCfEinImqK+Z5rSTSorjkN2OL7O/fY9lYol4k5PUhrEeChYxAjWRvI5ukR2v1S2Kxf1qutVkV2x7ZrjOjlxa965hxyj5CjDEs1+6b03SEgWU6EJx0p1HTvVwRLzQins2IvUzTFZIyHtGuowDFVwXR+7AydGmWAokSaEhrN1e8TUxwrNYlD0xljPVK/vVz8y+tmOqoHUybSTFNBFouijCOdoPxzNGCSEs0nhmAimbkVkRGWmGiT9GE8PUp+p+03Ipj+K1Xblwt4yjAMZzAGThQgwbcQBN8IMDgAZ7g2RLWo/VivS5aV6zlzBH8gPX2CdCyjgc=</latexit><latexit sha1_base64="Olr127mk9kZLbBuOIl3w31Dy+Ks=">AB7HicdVBNS8NAEJ34WetX1aOXxSJ4KkMbT0IRS8eK5i20Iay2W7apZtN2N0IpfQ3ePGgiFd/kDf/jZu2go+GHi8N8PMvDlTGnb/rBWVtfWNzYLW8Xtnd29/dLBYUslmSTUJwlPZCfEinImqK+Z5rSTSorjkN2OL7O/fY9lYol4k5PUhrEeChYxAjWRvI5ukR2v1S2Kxf1qutVkV2x7ZrjOjlxa965hxyj5CjDEs1+6b03SEgWU6EJx0p1HTvVwRLzQins2IvUzTFZIyHtGuowDFVwXR+7AydGmWAokSaEhrN1e8TUxwrNYlD0xljPVK/vVz8y+tmOqoHUybSTFNBFouijCOdoPxzNGCSEs0nhmAimbkVkRGWmGiT9GE8PUp+p+03Ipj+K1Xblwt4yjAMZzAGThQgwbcQBN8IMDgAZ7g2RLWo/VivS5aV6zlzBH8gPX2CdCyjgc=</latexit>

l = 1

<latexit sha1_base64="GgyRi2GthwqzaySoYPYFqHZiA=">AB7HicdVBNS8NAEJ34WetX1aOXxSJ4KkMbT0IRS8eK5i20Iay2W7apZtN2N0IpfQ3ePGgiFd/kDf/jZu2go+GHi8N8PMvDlTGnb/rBWVtfWNzYLW8Xtnd29/dLBYUslmSTUJwlPZCfEinImqK+Z5rSTSorjkN2OL7O/fY9lYol4k5PUhrEeChYxAjWRvI5ukROv1S2Kxf1qutVkV2x7ZrjOjlxa965hxyj5CjDEs1+6b03SEgWU6EJx0p1HTvVwRLzQins2IvUzTFZIyHtGuowDFVwXR+7AydGmWAokSaEhrN1e8TUxwrNYlD0xljPVK/vVz8y+tmOqoHUybSTFNBFouijCOdoPxzNGCSEs0nhmAimbkVkRGWmGiT9GE8PUp+p+03Ipj+K1Xblwt4yjAMZzAGThQgwbcQBN8IMDgAZ7g2RLWo/VivS5aV6zlzBH8gPX2CdI2jg=</latexit><latexit sha1_base64="GgyRi2GthwqzaySoYPYFqHZiA=">AB7HicdVBNS8NAEJ34WetX1aOXxSJ4KkMbT0IRS8eK5i20Iay2W7apZtN2N0IpfQ3ePGgiFd/kDf/jZu2go+GHi8N8PMvDlTGnb/rBWVtfWNzYLW8Xtnd29/dLBYUslmSTUJwlPZCfEinImqK+Z5rSTSorjkN2OL7O/fY9lYol4k5PUhrEeChYxAjWRvI5ukROv1S2Kxf1qutVkV2x7ZrjOjlxa965hxyj5CjDEs1+6b03SEgWU6EJx0p1HTvVwRLzQins2IvUzTFZIyHtGuowDFVwXR+7AydGmWAokSaEhrN1e8TUxwrNYlD0xljPVK/vVz8y+tmOqoHUybSTFNBFouijCOdoPxzNGCSEs0nhmAimbkVkRGWmGiT9GE8PUp+p+03Ipj+K1Xblwt4yjAMZzAGThQgwbcQBN8IMDgAZ7g2RLWo/VivS5aV6zlzBH8gPX2CdI2jg=</latexit><latexit sha1_base64="GgyRi2GthwqzaySoYPYFqHZiA=">AB7HicdVBNS8NAEJ34WetX1aOXxSJ4KkMbT0IRS8eK5i20Iay2W7apZtN2N0IpfQ3ePGgiFd/kDf/jZu2go+GHi8N8PMvDlTGnb/rBWVtfWNzYLW8Xtnd29/dLBYUslmSTUJwlPZCfEinImqK+Z5rSTSorjkN2OL7O/fY9lYol4k5PUhrEeChYxAjWRvI5ukROv1S2Kxf1qutVkV2x7ZrjOjlxa965hxyj5CjDEs1+6b03SEgWU6EJx0p1HTvVwRLzQins2IvUzTFZIyHtGuowDFVwXR+7AydGmWAokSaEhrN1e8TUxwrNYlD0xljPVK/vVz8y+tmOqoHUybSTFNBFouijCOdoPxzNGCSEs0nhmAimbkVkRGWmGiT9GE8PUp+p+03Ipj+K1Xblwt4yjAMZzAGThQgwbcQBN8IMDgAZ7g2RLWo/VivS5aV6zlzBH8gPX2CdI2jg=</latexit><latexit sha1_base64="GgyRi2GthwqzaySoYPYFqHZiA=">AB7HicdVBNS8NAEJ34WetX1aOXxSJ4KkMbT0IRS8eK5i20Iay2W7apZtN2N0IpfQ3ePGgiFd/kDf/jZu2go+GHi8N8PMvDlTGnb/rBWVtfWNzYLW8Xtnd29/dLBYUslmSTUJwlPZCfEinImqK+Z5rSTSorjkN2OL7O/fY9lYol4k5PUhrEeChYxAjWRvI5ukROv1S2Kxf1qutVkV2x7ZrjOjlxa965hxyj5CjDEs1+6b03SEgWU6EJx0p1HTvVwRLzQins2IvUzTFZIyHtGuowDFVwXR+7AydGmWAokSaEhrN1e8TUxwrNYlD0xljPVK/vVz8y+tmOqoHUybSTFNBFouijCOdoPxzNGCSEs0nhmAimbkVkRGWmGiT9GE8PUp+p+03Ipj+K1Xblwt4yjAMZzAGThQgwbcQBN8IMDgAZ7g2RLWo/VivS5aV6zlzBH8gPX2CdI2jg=</latexit>

. . .

<latexit sha1_base64="hsmkPBNcPgP5pZJP+ZcaC6imaV4=">AB7HicdVBNS8NAFNzUr1q/qh69LBbBU0liaOut6MVjBdMW2lA2027dLMJuy9CKf0NXjwo4tUf5M1/46atoKIDC8PMG/a9CVPBNdj2h1VYW9/Y3Cpul3Z29/YPyodHbZ1kijKfJiJR3ZBoJrhkPnAQrJsqRuJQsE4uc79zj1TmifyDqYpC2IykjzilICR/P4wAT0oV+zqZaPmejVsV27rhOTty6d+Fhxyg5KmiF1qD8bnI0i5kEKojWPcdOIZgRBZwKNi/1M81SQidkxHqGShIzHcwWy87xmVGOEqUeRLwQv2emJFY62kcmsmYwFj/9nLxL6+XQdQIZlymGTBJlx9FmcCQ4PxyPOSKURBTQwhV3OyK6ZgoQsH0UzIlfF2K/ydt+oYfutVmlerOoroBJ2ic+SgOmqiG9RCPqKIowf0hJ4taT1aL9brcrRgrTLH6Aest09RiI8E</latexit><latexit sha1_base64="hsmkPBNcPgP5pZJP+ZcaC6imaV4=">AB7HicdVBNS8NAFNzUr1q/qh69LBbBU0liaOut6MVjBdMW2lA2027dLMJuy9CKf0NXjwo4tUf5M1/46atoKIDC8PMG/a9CVPBNdj2h1VYW9/Y3Cpul3Z29/YPyodHbZ1kijKfJiJR3ZBoJrhkPnAQrJsqRuJQsE4uc79zj1TmifyDqYpC2IykjzilICR/P4wAT0oV+zqZaPmejVsV27rhOTty6d+Fhxyg5KmiF1qD8bnI0i5kEKojWPcdOIZgRBZwKNi/1M81SQidkxHqGShIzHcwWy87xmVGOEqUeRLwQv2emJFY62kcmsmYwFj/9nLxL6+XQdQIZlymGTBJlx9FmcCQ4PxyPOSKURBTQwhV3OyK6ZgoQsH0UzIlfF2K/ydt+oYfutVmlerOoroBJ2ic+SgOmqiG9RCPqKIowf0hJ4taT1aL9brcrRgrTLH6Aest09RiI8E</latexit><latexit sha1_base64="hsmkPBNcPgP5pZJP+ZcaC6imaV4=">AB7HicdVBNS8NAFNzUr1q/qh69LBbBU0liaOut6MVjBdMW2lA2027dLMJuy9CKf0NXjwo4tUf5M1/46atoKIDC8PMG/a9CVPBNdj2h1VYW9/Y3Cpul3Z29/YPyodHbZ1kijKfJiJR3ZBoJrhkPnAQrJsqRuJQsE4uc79zj1TmifyDqYpC2IykjzilICR/P4wAT0oV+zqZaPmejVsV27rhOTty6d+Fhxyg5KmiF1qD8bnI0i5kEKojWPcdOIZgRBZwKNi/1M81SQidkxHqGShIzHcwWy87xmVGOEqUeRLwQv2emJFY62kcmsmYwFj/9nLxL6+XQdQIZlymGTBJlx9FmcCQ4PxyPOSKURBTQwhV3OyK6ZgoQsH0UzIlfF2K/ydt+oYfutVmlerOoroBJ2ic+SgOmqiG9RCPqKIowf0hJ4taT1aL9brcrRgrTLH6Aest09RiI8E</latexit><latexit sha1_base64="hsmkPBNcPgP5pZJP+ZcaC6imaV4=">AB7HicdVBNS8NAFNzUr1q/qh69LBbBU0liaOut6MVjBdMW2lA2027dLMJuy9CKf0NXjwo4tUf5M1/46atoKIDC8PMG/a9CVPBNdj2h1VYW9/Y3Cpul3Z29/YPyodHbZ1kijKfJiJR3ZBoJrhkPnAQrJsqRuJQsE4uc79zj1TmifyDqYpC2IykjzilICR/P4wAT0oV+zqZaPmejVsV27rhOTty6d+Fhxyg5KmiF1qD8bnI0i5kEKojWPcdOIZgRBZwKNi/1M81SQidkxHqGShIzHcwWy87xmVGOEqUeRLwQv2emJFY62kcmsmYwFj/9nLxL6+XQdQIZlymGTBJlx9FmcCQ4PxyPOSKURBTQwhV3OyK6ZgoQsH0UzIlfF2K/ydt+oYfutVmlerOoroBJ2ic+SgOmqiG9RCPqKIowf0hJ4taT1aL9brcrRgrTLH6Aest09RiI8E</latexit>

φl : R → R

<latexit sha1_base64="ESQuNY1HTyCY/1JXMW13Kv1bZX0=">ACnicdVDLSgMxFM34rPU16tJNtAiuSlLEtq6KblxWsQ/ojCWTZtrQzIMkI5Shazf+ihsXirj1C9z5N2baClX0QODknHu59x4vFlxphD6thcWl5ZXV3Fp+fWNza9ve2W2qKJGUNWgkItn2iGKCh6yhuRasHUtGAk+wlje8yPzWHZOKR+GNHsXMDUg/5D6nRBupax848YDfijPoBEQPC+9HkNHR3Pfrl1ARYQxhmBJdPkSHVaqWEKxBnlkEBzFDv2h9OL6JwEJNBVGqg1Gs3ZRIzalg47yTKBYTOiR91jE0JAFTbjo5ZQyPjNKDfiTNCzWcqPMdKQmUGgWeqcw2VL+9TPzL6yTar7gpD+NEs5BOB/mJgObWLBfY45JRLUaGECq52RXSAZGEapNe3oTwfSn8nzRLRWz41Umhdj6LIwf2wSE4BhiUQ1cgjpoAruwSN4Bi/Wg/VkvVpv09IFa9azB37Aev8CKY+akw=</latexit><latexit sha1_base64="ESQuNY1HTyCY/1JXMW13Kv1bZX0=">ACnicdVDLSgMxFM34rPU16tJNtAiuSlLEtq6KblxWsQ/ojCWTZtrQzIMkI5Shazf+ihsXirj1C9z5N2baClX0QODknHu59x4vFlxphD6thcWl5ZXV3Fp+fWNza9ve2W2qKJGUNWgkItn2iGKCh6yhuRasHUtGAk+wlje8yPzWHZOKR+GNHsXMDUg/5D6nRBupax848YDfijPoBEQPC+9HkNHR3Pfrl1ARYQxhmBJdPkSHVaqWEKxBnlkEBzFDv2h9OL6JwEJNBVGqg1Gs3ZRIzalg47yTKBYTOiR91jE0JAFTbjo5ZQyPjNKDfiTNCzWcqPMdKQmUGgWeqcw2VL+9TPzL6yTar7gpD+NEs5BOB/mJgObWLBfY45JRLUaGECq52RXSAZGEapNe3oTwfSn8nzRLRWz41Umhdj6LIwf2wSE4BhiUQ1cgjpoAruwSN4Bi/Wg/VkvVpv09IFa9azB37Aev8CKY+akw=</latexit><latexit sha1_base64="ESQuNY1HTyCY/1JXMW13Kv1bZX0=">ACnicdVDLSgMxFM34rPU16tJNtAiuSlLEtq6KblxWsQ/ojCWTZtrQzIMkI5Shazf+ihsXirj1C9z5N2baClX0QODknHu59x4vFlxphD6thcWl5ZXV3Fp+fWNza9ve2W2qKJGUNWgkItn2iGKCh6yhuRasHUtGAk+wlje8yPzWHZOKR+GNHsXMDUg/5D6nRBupax848YDfijPoBEQPC+9HkNHR3Pfrl1ARYQxhmBJdPkSHVaqWEKxBnlkEBzFDv2h9OL6JwEJNBVGqg1Gs3ZRIzalg47yTKBYTOiR91jE0JAFTbjo5ZQyPjNKDfiTNCzWcqPMdKQmUGgWeqcw2VL+9TPzL6yTar7gpD+NEs5BOB/mJgObWLBfY45JRLUaGECq52RXSAZGEapNe3oTwfSn8nzRLRWz41Umhdj6LIwf2wSE4BhiUQ1cgjpoAruwSN4Bi/Wg/VkvVpv09IFa9azB37Aev8CKY+akw=</latexit><latexit sha1_base64="ESQuNY1HTyCY/1JXMW13Kv1bZX0=">ACnicdVDLSgMxFM34rPU16tJNtAiuSlLEtq6KblxWsQ/ojCWTZtrQzIMkI5Shazf+ihsXirj1C9z5N2baClX0QODknHu59x4vFlxphD6thcWl5ZXV3Fp+fWNza9ve2W2qKJGUNWgkItn2iGKCh6yhuRasHUtGAk+wlje8yPzWHZOKR+GNHsXMDUg/5D6nRBupax848YDfijPoBEQPC+9HkNHR3Pfrl1ARYQxhmBJdPkSHVaqWEKxBnlkEBzFDv2h9OL6JwEJNBVGqg1Gs3ZRIzalg47yTKBYTOiR91jE0JAFTbjo5ZQyPjNKDfiTNCzWcqPMdKQmUGgWeqcw2VL+9TPzL6yTar7gpD+NEs5BOB/mJgObWLBfY45JRLUaGECq52RXSAZGEapNe3oTwfSn8nzRLRWz41Umhdj6LIwf2wSE4BhiUQ1cgjpoAruwSN4Bi/Wg/VkvVpv09IFa9azB37Aev8CKY+akw=</latexit>

W l

αβ, bl α

<latexit sha1_base64="iuahqj6ZeaomXxwZP5x0kCdv14=">ACXicdZDLSsNAFIYnXmu9RV26GSyCymJjRh3RTcuK9gLNGk4mU7boZMLMxOhG7d+CpuXCji1jdw59s4vQgq+sPAz3fO4cz5w5QzqSzrw1hYXFpeWS2sFdc3Nre2zZ3dhkwyQWidJDwRrRAk5SymdcUp61UIhCTpvh8HJSb95SIVkS36hRSv0I+jHrMQJKo8DEzSD3gKcDwF5IFYw7/Bh7OAxmsMDs2SVLdepuDa2yqeVc9upaGM5rqOJXbamKqG5aoH57nUTkU0VoSDlG3bSpWfg1CMcDouepmkKZAh9Glb2xgiKv18eskYH2rSxb1E6BcrPKXfJ3KIpBxFoe6MQA3k79oE/lVrZ6rn+jmL0zRmMwW9TKOVYInseAuE5QoPtIGiGD6r5gMQABROryiDuHrUvy/aZyUbe2vnVL1Yh5HAe2jA3SEbHSGqugK1VAdEXSHtATejbujUfjxXidtS4Y85k9EPG2yfw5nc</latexit><latexit sha1_base64="iuahqj6ZeaomXxwZP5x0kCdv14=">ACXicdZDLSsNAFIYnXmu9RV26GSyCymJjRh3RTcuK9gLNGk4mU7boZMLMxOhG7d+CpuXCji1jdw59s4vQgq+sPAz3fO4cz5w5QzqSzrw1hYXFpeWS2sFdc3Nre2zZ3dhkwyQWidJDwRrRAk5SymdcUp61UIhCTpvh8HJSb95SIVkS36hRSv0I+jHrMQJKo8DEzSD3gKcDwF5IFYw7/Bh7OAxmsMDs2SVLdepuDa2yqeVc9upaGM5rqOJXbamKqG5aoH57nUTkU0VoSDlG3bSpWfg1CMcDouepmkKZAh9Glb2xgiKv18eskYH2rSxb1E6BcrPKXfJ3KIpBxFoe6MQA3k79oE/lVrZ6rn+jmL0zRmMwW9TKOVYInseAuE5QoPtIGiGD6r5gMQABROryiDuHrUvy/aZyUbe2vnVL1Yh5HAe2jA3SEbHSGqugK1VAdEXSHtATejbujUfjxXidtS4Y85k9EPG2yfw5nc</latexit><latexit sha1_base64="iuahqj6ZeaomXxwZP5x0kCdv14=">ACXicdZDLSsNAFIYnXmu9RV26GSyCymJjRh3RTcuK9gLNGk4mU7boZMLMxOhG7d+CpuXCji1jdw59s4vQgq+sPAz3fO4cz5w5QzqSzrw1hYXFpeWS2sFdc3Nre2zZ3dhkwyQWidJDwRrRAk5SymdcUp61UIhCTpvh8HJSb95SIVkS36hRSv0I+jHrMQJKo8DEzSD3gKcDwF5IFYw7/Bh7OAxmsMDs2SVLdepuDa2yqeVc9upaGM5rqOJXbamKqG5aoH57nUTkU0VoSDlG3bSpWfg1CMcDouepmkKZAh9Glb2xgiKv18eskYH2rSxb1E6BcrPKXfJ3KIpBxFoe6MQA3k79oE/lVrZ6rn+jmL0zRmMwW9TKOVYInseAuE5QoPtIGiGD6r5gMQABROryiDuHrUvy/aZyUbe2vnVL1Yh5HAe2jA3SEbHSGqugK1VAdEXSHtATejbujUfjxXidtS4Y85k9EPG2yfw5nc</latexit><latexit sha1_base64="iuahqj6ZeaomXxwZP5x0kCdv14=">ACXicdZDLSsNAFIYnXmu9RV26GSyCymJjRh3RTcuK9gLNGk4mU7boZMLMxOhG7d+CpuXCji1jdw59s4vQgq+sPAz3fO4cz5w5QzqSzrw1hYXFpeWS2sFdc3Nre2zZ3dhkwyQWidJDwRrRAk5SymdcUp61UIhCTpvh8HJSb95SIVkS36hRSv0I+jHrMQJKo8DEzSD3gKcDwF5IFYw7/Bh7OAxmsMDs2SVLdepuDa2yqeVc9upaGM5rqOJXbamKqG5aoH57nUTkU0VoSDlG3bSpWfg1CMcDouepmkKZAh9Glb2xgiKv18eskYH2rSxb1E6BcrPKXfJ3KIpBxFoe6MQA3k79oE/lVrZ6rn+jmL0zRmMwW9TKOVYInseAuE5QoPtIGiGD6r5gMQABROryiDuHrUvy/aZyUbe2vnVL1Yh5HAe2jA3SEbHSGqugK1VAdEXSHtATejbujUfjxXidtS4Y85k9EPG2yfw5nc</latexit>

{

<latexit sha1_base64="swUn7zWqnVi68t3BIo8OZjaVI=">AB6XicdVDLSgNBEOz1GeMr6tHLYBA8hVmz4noLevEYxTwgWcLsZDYZMju7zMwKYckfePGgiFf/yJt/4+QhqGhBQ1HVTXdXmAquDcYfztLyuraemGjuLm1vbNb2tv6iRTlDVoIhLVDolmgkvWMNwI1k4VI3EoWCscXU391j1TmifyzoxTFsRkIHnEKTFWu3mvVIZV7DvVX0X4cpZ9cL1qpZgz/es4lbwDGVYoN4rvXf7Cc1iJg0VROuOi1MT5EQZTgWbFLuZimhIzJgHUsliZkO8tmlE3RslT6KEmVLGjRTv0/kJNZ6HIe2MyZmqH97U/Evr5OZyA9yLtPMEni6JMIJOg6duozxWjRowtIVRxeyuiQ6INTacog3h61P0P2meVlzLb7xy7XIRwEO4QhOwIVzqME1KEBFCJ4gCd4dkbOo/PivM5bl5zFzAH8gP2CQakja8=</latexit><latexit sha1_base64="swUn7zWqnVi68t3BIo8OZjaVI=">AB6XicdVDLSgNBEOz1GeMr6tHLYBA8hVmz4noLevEYxTwgWcLsZDYZMju7zMwKYckfePGgiFf/yJt/4+QhqGhBQ1HVTXdXmAquDcYfztLyuraemGjuLm1vbNb2tv6iRTlDVoIhLVDolmgkvWMNwI1k4VI3EoWCscXU391j1TmifyzoxTFsRkIHnEKTFWu3mvVIZV7DvVX0X4cpZ9cL1qpZgz/es4lbwDGVYoN4rvXf7Cc1iJg0VROuOi1MT5EQZTgWbFLuZimhIzJgHUsliZkO8tmlE3RslT6KEmVLGjRTv0/kJNZ6HIe2MyZmqH97U/Evr5OZyA9yLtPMEni6JMIJOg6duozxWjRowtIVRxeyuiQ6INTacog3h61P0P2meVlzLb7xy7XIRwEO4QhOwIVzqME1KEBFCJ4gCd4dkbOo/PivM5bl5zFzAH8gP2CQakja8=</latexit><latexit sha1_base64="swUn7zWqnVi68t3BIo8OZjaVI=">AB6XicdVDLSgNBEOz1GeMr6tHLYBA8hVmz4noLevEYxTwgWcLsZDYZMju7zMwKYckfePGgiFf/yJt/4+QhqGhBQ1HVTXdXmAquDcYfztLyuraemGjuLm1vbNb2tv6iRTlDVoIhLVDolmgkvWMNwI1k4VI3EoWCscXU391j1TmifyzoxTFsRkIHnEKTFWu3mvVIZV7DvVX0X4cpZ9cL1qpZgz/es4lbwDGVYoN4rvXf7Cc1iJg0VROuOi1MT5EQZTgWbFLuZimhIzJgHUsliZkO8tmlE3RslT6KEmVLGjRTv0/kJNZ6HIe2MyZmqH97U/Evr5OZyA9yLtPMEni6JMIJOg6duozxWjRowtIVRxeyuiQ6INTacog3h61P0P2meVlzLb7xy7XIRwEO4QhOwIVzqME1KEBFCJ4gCd4dkbOo/PivM5bl5zFzAH8gP2CQakja8=</latexit><latexit sha1_base64="swUn7zWqnVi68t3BIo8OZjaVI=">AB6XicdVDLSgNBEOz1GeMr6tHLYBA8hVmz4noLevEYxTwgWcLsZDYZMju7zMwKYckfePGgiFf/yJt/4+QhqGhBQ1HVTXdXmAquDcYfztLyuraemGjuLm1vbNb2tv6iRTlDVoIhLVDolmgkvWMNwI1k4VI3EoWCscXU391j1TmifyzoxTFsRkIHnEKTFWu3mvVIZV7DvVX0X4cpZ9cL1qpZgz/es4lbwDGVYoN4rvXf7Cc1iJg0VROuOi1MT5EQZTgWbFLuZimhIzJgHUsliZkO8tmlE3RslT6KEmVLGjRTv0/kJNZ6HIe2MyZmqH97U/Evr5OZyA9yLtPMEni6JMIJOg6duozxWjRowtIVRxeyuiQ6INTacog3h61P0P2meVlzLb7xy7XIRwEO4QhOwIVzqME1KEBFCJ4gCd4dkbOo/PivM5bl5zFzAH8gP2CQakja8=</latexit>

<latexit sha1_base64="0rzZX0DIjOIJMh6qrh7Vjf51F4g=">AB6nicdZBNS8NAEIYn9avWr6pHL4tF8FRSUfRY9OKxov2ANpTNdtIu3WzC7kYsoT/BiwdFvPqLvPlv3LQpqOgLCy/PzLAzrx8Lro3rfjqFpeWV1bXiemljc2t7p7y719JRohg2WSQi1fGpRsElNg03AjuxQhr6Atv+Cqrt+9RaR7JOzOJ0QvpUPKAM2osun3o83654lbP3EzErboLk5NaTiqQq9Evf/QGEUtClIYJqnW35sbGS6kynAmclnqJxpiyMR1i1pJQ9ReOlt1So4sGZAgUvZJQ2b0+0RKQ60noW87Q2pG+nctg3/VuokJLryUyzgxKNn8oyARxEQku5sMuEJmxMQayhS3uxI2oyY9Mp2RAWl5L/TeukWrP+5rRSv8zjKMIBHMIx1OAc6nANDWgCgyE8wjO8OMJ5cl6dt3lrwcln9uGHnPcvbySN4g=</latexit><latexit sha1_base64="0rzZX0DIjOIJMh6qrh7Vjf51F4g=">AB6nicdZBNS8NAEIYn9avWr6pHL4tF8FRSUfRY9OKxov2ANpTNdtIu3WzC7kYsoT/BiwdFvPqLvPlv3LQpqOgLCy/PzLAzrx8Lro3rfjqFpeWV1bXiemljc2t7p7y719JRohg2WSQi1fGpRsElNg03AjuxQhr6Atv+Cqrt+9RaR7JOzOJ0QvpUPKAM2osun3o83654lbP3EzErboLk5NaTiqQq9Evf/QGEUtClIYJqnW35sbGS6kynAmclnqJxpiyMR1i1pJQ9ReOlt1So4sGZAgUvZJQ2b0+0RKQ60noW87Q2pG+nctg3/VuokJLryUyzgxKNn8oyARxEQku5sMuEJmxMQayhS3uxI2oyY9Mp2RAWl5L/TeukWrP+5rRSv8zjKMIBHMIx1OAc6nANDWgCgyE8wjO8OMJ5cl6dt3lrwcln9uGHnPcvbySN4g=</latexit><latexit sha1_base64="0rzZX0DIjOIJMh6qrh7Vjf51F4g=">AB6nicdZBNS8NAEIYn9avWr6pHL4tF8FRSUfRY9OKxov2ANpTNdtIu3WzC7kYsoT/BiwdFvPqLvPlv3LQpqOgLCy/PzLAzrx8Lro3rfjqFpeWV1bXiemljc2t7p7y719JRohg2WSQi1fGpRsElNg03AjuxQhr6Atv+Cqrt+9RaR7JOzOJ0QvpUPKAM2osun3o83654lbP3EzErboLk5NaTiqQq9Evf/QGEUtClIYJqnW35sbGS6kynAmclnqJxpiyMR1i1pJQ9ReOlt1So4sGZAgUvZJQ2b0+0RKQ60noW87Q2pG+nctg3/VuokJLryUyzgxKNn8oyARxEQku5sMuEJmxMQayhS3uxI2oyY9Mp2RAWl5L/TeukWrP+5rRSv8zjKMIBHMIx1OAc6nANDWgCgyE8wjO8OMJ5cl6dt3lrwcln9uGHnPcvbySN4g=</latexit><latexit sha1_base64="0rzZX0DIjOIJMh6qrh7Vjf51F4g=">AB6nicdZBNS8NAEIYn9avWr6pHL4tF8FRSUfRY9OKxov2ANpTNdtIu3WzC7kYsoT/BiwdFvPqLvPlv3LQpqOgLCy/PzLAzrx8Lro3rfjqFpeWV1bXiemljc2t7p7y719JRohg2WSQi1fGpRsElNg03AjuxQhr6Atv+Cqrt+9RaR7JOzOJ0QvpUPKAM2osun3o83654lbP3EzErboLk5NaTiqQq9Evf/QGEUtClIYJqnW35sbGS6kynAmclnqJxpiyMR1i1pJQ9ReOlt1So4sGZAgUvZJQ2b0+0RKQ60noW87Q2pG+nctg3/VuokJLryUyzgxKNn8oyARxEQku5sMuEJmxMQayhS3uxI2oyY9Mp2RAWl5L/TeukWrP+5rRSv8zjKMIBHMIx1OAc6nANDWgCgyE8wjO8OMJ5cl6dt3lrwcln9uGHnPcvbySN4g=</latexit>

(i ∈ M)

<latexit sha1_base64="PIsC7W+8F9drz72+G1ptSCbKBiQ=">AB/HicdVDLSsNAFL3xWesr2qWbwSLUTUlF0WXRjRuhgn1AE8pkOmHTiZhZiKEUH/FjQtF3Poh7vwbJ20Knpg4HDOvdwzx485U9pxPq2l5ZXVtfXSRnlza3tn197b76gokYS2ScQj2fOxopwJ2tZMc9qLJcWhz2nXn1zlfveSsUicafTmHohHgkWMIK1kQZ2pcaQywRyQ6zHBPsZno8sKtO/czJgZy6syCF0iUKhRoDewPdxiRJKRCE46V6jecWHsZlpoRTqdlN1E0xmSCR7RvqMAhV42Cz9FR0YZoiCS5gmNZur3jQyHSqWhbybzjOq3l4t/ef1EBxdexkScaCrI/FCQcKQjlDeBhkxSonlqCaSmayIjLHERJu+yqaExU/R/6RzUm8YfntabV4WdZTgA6hBg04hyZcQwvaQCFR3iGF+vBerJerbf56JV7FTgB6z3L6BClBs=</latexit><latexit sha1_base64="PIsC7W+8F9drz72+G1ptSCbKBiQ=">AB/HicdVDLSsNAFL3xWesr2qWbwSLUTUlF0WXRjRuhgn1AE8pkOmHTiZhZiKEUH/FjQtF3Poh7vwbJ20Knpg4HDOvdwzx485U9pxPq2l5ZXVtfXSRnlza3tn197b76gokYS2ScQj2fOxopwJ2tZMc9qLJcWhz2nXn1zlfveSsUicafTmHohHgkWMIK1kQZ2pcaQywRyQ6zHBPsZno8sKtO/czJgZy6syCF0iUKhRoDewPdxiRJKRCE46V6jecWHsZlpoRTqdlN1E0xmSCR7RvqMAhV42Cz9FR0YZoiCS5gmNZur3jQyHSqWhbybzjOq3l4t/ef1EBxdexkScaCrI/FCQcKQjlDeBhkxSonlqCaSmayIjLHERJu+yqaExU/R/6RzUm8YfntabV4WdZTgA6hBg04hyZcQwvaQCFR3iGF+vBerJerbf56JV7FTgB6z3L6BClBs=</latexit><latexit sha1_base64="PIsC7W+8F9drz72+G1ptSCbKBiQ=">AB/HicdVDLSsNAFL3xWesr2qWbwSLUTUlF0WXRjRuhgn1AE8pkOmHTiZhZiKEUH/FjQtF3Poh7vwbJ20Knpg4HDOvdwzx485U9pxPq2l5ZXVtfXSRnlza3tn197b76gokYS2ScQj2fOxopwJ2tZMc9qLJcWhz2nXn1zlfveSsUicafTmHohHgkWMIK1kQZ2pcaQywRyQ6zHBPsZno8sKtO/czJgZy6syCF0iUKhRoDewPdxiRJKRCE46V6jecWHsZlpoRTqdlN1E0xmSCR7RvqMAhV42Cz9FR0YZoiCS5gmNZur3jQyHSqWhbybzjOq3l4t/ef1EBxdexkScaCrI/FCQcKQjlDeBhkxSonlqCaSmayIjLHERJu+yqaExU/R/6RzUm8YfntabV4WdZTgA6hBg04hyZcQwvaQCFR3iGF+vBerJerbf56JV7FTgB6z3L6BClBs=</latexit><latexit sha1_base64="PIsC7W+8F9drz72+G1ptSCbKBiQ=">AB/HicdVDLSsNAFL3xWesr2qWbwSLUTUlF0WXRjRuhgn1AE8pkOmHTiZhZiKEUH/FjQtF3Poh7vwbJ20Knpg4HDOvdwzx485U9pxPq2l5ZXVtfXSRnlza3tn197b76gokYS2ScQj2fOxopwJ2tZMc9qLJcWhz2nXn1zlfveSsUicafTmHohHgkWMIK1kQZ2pcaQywRyQ6zHBPsZno8sKtO/czJgZy6syCF0iUKhRoDewPdxiRJKRCE46V6jecWHsZlpoRTqdlN1E0xmSCR7RvqMAhV42Cz9FR0YZoiCS5gmNZur3jQyHSqWhbybzjOq3l4t/ef1EBxdexkScaCrI/FCQcKQjlDeBhkxSonlqCaSmayIjLHERJu+yqaExU/R/6RzUm8YfntabV4WdZTgA6hBg04hyZcQwvaQCFR3iGF+vBerJerbf56JV7FTgB6z3L6BClBs=</latexit>

target

<latexit sha1_base64="+SeC0eZeOvMRhrfOgIpWUPEIvg=">AB83icdZBNS8NAEIYn9avWr6pHL4tF8FRSUfRY9OKxgv2AJpTNdtMu3WzC7kQoX/DiwdFvPpnvPlv3LQpqOgLCy/PzDCzb5BIYdB1P53Syura+kZ5s7K1vbO7V90/6Jg41Yy3WSxj3Quo4VIo3kaBkvcSzWkUSN4NJjd5vfvAtRGxusdpwv2IjpQIBaNokZd5OiJI9YjbFCtufULNxdx6+7SFKRkBoUag2qH94wZmnEFTJjek3AT9jGoUTPJZxUsNTyib0BHvW6toxI2fzW+ekRNLhiSMtX0KyZx+n8hoZMw0CmxnRHFsftdy+Fetn2J45WdCJSlyxRaLwlQSjEkeABkKzRnKqTWUaWFvJWxMNWVoY6rYEJY/Jf+bzlm9Yf3dea15XcRhiM4hlNowCU04RZa0AYGCTzCM7w4qfPkvDpvi9aSU8wcwg85719KGZHW</latexit><latexit sha1_base64="+SeC0eZeOvMRhrfOgIpWUPEIvg=">AB83icdZBNS8NAEIYn9avWr6pHL4tF8FRSUfRY9OKxgv2AJpTNdtMu3WzC7kQoX/DiwdFvPpnvPlv3LQpqOgLCy/PzDCzb5BIYdB1P53Syura+kZ5s7K1vbO7V90/6Jg41Yy3WSxj3Quo4VIo3kaBkvcSzWkUSN4NJjd5vfvAtRGxusdpwv2IjpQIBaNokZd5OiJI9YjbFCtufULNxdx6+7SFKRkBoUag2qH94wZmnEFTJjek3AT9jGoUTPJZxUsNTyib0BHvW6toxI2fzW+ekRNLhiSMtX0KyZx+n8hoZMw0CmxnRHFsftdy+Fetn2J45WdCJSlyxRaLwlQSjEkeABkKzRnKqTWUaWFvJWxMNWVoY6rYEJY/Jf+bzlm9Yf3dea15XcRhiM4hlNowCU04RZa0AYGCTzCM7w4qfPkvDpvi9aSU8wcwg85719KGZHW</latexit><latexit sha1_base64="+SeC0eZeOvMRhrfOgIpWUPEIvg=">AB83icdZBNS8NAEIYn9avWr6pHL4tF8FRSUfRY9OKxgv2AJpTNdtMu3WzC7kQoX/DiwdFvPpnvPlv3LQpqOgLCy/PzDCzb5BIYdB1P53Syura+kZ5s7K1vbO7V90/6Jg41Yy3WSxj3Quo4VIo3kaBkvcSzWkUSN4NJjd5vfvAtRGxusdpwv2IjpQIBaNokZd5OiJI9YjbFCtufULNxdx6+7SFKRkBoUag2qH94wZmnEFTJjek3AT9jGoUTPJZxUsNTyib0BHvW6toxI2fzW+ekRNLhiSMtX0KyZx+n8hoZMw0CmxnRHFsftdy+Fetn2J45WdCJSlyxRaLwlQSjEkeABkKzRnKqTWUaWFvJWxMNWVoY6rYEJY/Jf+bzlm9Yf3dea15XcRhiM4hlNowCU04RZa0AYGCTzCM7w4qfPkvDpvi9aSU8wcwg85719KGZHW</latexit><latexit sha1_base64="+SeC0eZeOvMRhrfOgIpWUPEIvg=">AB83icdZBNS8NAEIYn9avWr6pHL4tF8FRSUfRY9OKxgv2AJpTNdtMu3WzC7kQoX/DiwdFvPpnvPlv3LQpqOgLCy/PzDCzb5BIYdB1P53Syura+kZ5s7K1vbO7V90/6Jg41Yy3WSxj3Quo4VIo3kaBkvcSzWkUSN4NJjd5vfvAtRGxusdpwv2IjpQIBaNokZd5OiJI9YjbFCtufULNxdx6+7SFKRkBoUag2qH94wZmnEFTJjek3AT9jGoUTPJZxUsNTyib0BHvW6toxI2fzW+ekRNLhiSMtX0KyZx+n8hoZMw0CmxnRHFsftdy+Fetn2J45WdCJSlyxRaLwlQSjEkeABkKzRnKqTWUaWFvJWxMNWVoY6rYEJY/Jf+bzlm9Yf3dea15XcRhiM4hlNowCU04RZa0AYGCTzCM7w4qfPkvDpvi9aSU8wcwg85719KGZHW</latexit>

f(xi) : RN0 → RNL

<latexit sha1_base64="NkrVmKf7SZa9c1Dl3ReMomIl/0=">ACFnicdZDLSsNAFIYn9VbrLerSzWAR6sKSiK4KrpxIVLFXqCJYTKdtEMnF2YmYgl5Cje+ihsXirgVd76NkzYFLfrDwM93zmHO+d2IUSEN40srzMzOzS8UF0tLyura/r6RlOEMcekgUMW8raLBGE0IA1JSPtiBPku4y03MFZVm/dES5oGNzIYURsH/UC6lGMpEKOvudV7h26ewItH8m+6ybX6W1y6RgptGQ4BS9SRy8b1UMjEzSqxsTkxMxJGeSqO/qn1Q1x7JNAYoaE6JhGJO0EcUkxI2nJigWJEB6gHukoGyCfCDsZnZXCHUW60Au5eoGEI/pzIkG+EPfVZ3ZnmK6lsG/ap1Yesd2QoMoliTA4+8mEF1cZYR7FJOsGRDZRDmVO0KcR9xhKVKsqRCmFwK/zfN/aqp/NVBuXax1EW2AbVIAJjkANnIM6aAMHsATeAGv2qP2rL1p7+PWgpbPbIJf0j6+AXGpnvE=</latexit><latexit sha1_base64="NkrVmKf7SZa9c1Dl3ReMomIl/0=">ACFnicdZDLSsNAFIYn9VbrLerSzWAR6sKSiK4KrpxIVLFXqCJYTKdtEMnF2YmYgl5Cje+ihsXirgVd76NkzYFLfrDwM93zmHO+d2IUSEN40srzMzOzS8UF0tLyura/r6RlOEMcekgUMW8raLBGE0IA1JSPtiBPku4y03MFZVm/dES5oGNzIYURsH/UC6lGMpEKOvudV7h26ewItH8m+6ybX6W1y6RgptGQ4BS9SRy8b1UMjEzSqxsTkxMxJGeSqO/qn1Q1x7JNAYoaE6JhGJO0EcUkxI2nJigWJEB6gHukoGyCfCDsZnZXCHUW60Au5eoGEI/pzIkG+EPfVZ3ZnmK6lsG/ap1Yesd2QoMoliTA4+8mEF1cZYR7FJOsGRDZRDmVO0KcR9xhKVKsqRCmFwK/zfN/aqp/NVBuXax1EW2AbVIAJjkANnIM6aAMHsATeAGv2qP2rL1p7+PWgpbPbIJf0j6+AXGpnvE=</latexit><latexit sha1_base64="NkrVmKf7SZa9c1Dl3ReMomIl/0=">ACFnicdZDLSsNAFIYn9VbrLerSzWAR6sKSiK4KrpxIVLFXqCJYTKdtEMnF2YmYgl5Cje+ihsXirgVd76NkzYFLfrDwM93zmHO+d2IUSEN40srzMzOzS8UF0tLyura/r6RlOEMcekgUMW8raLBGE0IA1JSPtiBPku4y03MFZVm/dES5oGNzIYURsH/UC6lGMpEKOvudV7h26ewItH8m+6ybX6W1y6RgptGQ4BS9SRy8b1UMjEzSqxsTkxMxJGeSqO/qn1Q1x7JNAYoaE6JhGJO0EcUkxI2nJigWJEB6gHukoGyCfCDsZnZXCHUW60Au5eoGEI/pzIkG+EPfVZ3ZnmK6lsG/ap1Yesd2QoMoliTA4+8mEF1cZYR7FJOsGRDZRDmVO0KcR9xhKVKsqRCmFwK/zfN/aqp/NVBuXax1EW2AbVIAJjkANnIM6aAMHsATeAGv2qP2rL1p7+PWgpbPbIJf0j6+AXGpnvE=</latexit><latexit sha1_base64="NkrVmKf7SZa9c1Dl3ReMomIl/0=">ACFnicdZDLSsNAFIYn9VbrLerSzWAR6sKSiK4KrpxIVLFXqCJYTKdtEMnF2YmYgl5Cje+ihsXirgVd76NkzYFLfrDwM93zmHO+d2IUSEN40srzMzOzS8UF0tLyura/r6RlOEMcekgUMW8raLBGE0IA1JSPtiBPku4y03MFZVm/dES5oGNzIYURsH/UC6lGMpEKOvudV7h26ewItH8m+6ybX6W1y6RgptGQ4BS9SRy8b1UMjEzSqxsTkxMxJGeSqO/qn1Q1x7JNAYoaE6JhGJO0EcUkxI2nJigWJEB6gHukoGyCfCDsZnZXCHUW60Au5eoGEI/pzIkG+EPfVZ3ZnmK6lsG/ap1Yesd2QoMoliTA4+8mEF1cZYR7FJOsGRDZRDmVO0KcR9xhKVKsqRCmFwK/zfN/aqp/NVBuXax1EW2AbVIAJjkANnIM6aAMHsATeAGv2qP2rL1p7+PWgpbPbIJf0j6+AXGpnvE=</latexit>

l = L

<latexit sha1_base64="DHtV0M4TMijiq/4AbB2ReXOko4=">AB7HicdVBNS8NAEJ3Ur1q/qh69LBbBU0liaOtBKHrx4KGC/YA2lM12y7dbMLuRihv8GLB0W8+oO8+W/ctBVU9MHA470ZuYFMWdK2/aHlVtZXVvfyG8WtrZ3dveK+wctFSWS0CaJeCQ7AVaUM0GbmlO7GkOAw4bQeTq8xv31OpWCTu9DSmfohHg0ZwdpITY4u0E2/WL57WK61WQXbtquM6GXGr3pmHKNkKMESjX7xvTeISBJSoQnHSnUdO9Z+iqVmhNZoZcoGmMywSPaNVTgkCo/nR87QydGaBhJE0Jjebq94kUh0pNw8B0hliP1W8vE/yuoke1vyUiTjRVJDFomHCkY5Q9jkaMEmJ5lNDMJHM3IrIGEtMtMmnYEL4+hT9T1pu2TH81ivVL5dx5OEIjuEUHKhCHa6hAU0gwOABnuDZEtaj9WK9Llpz1nLmEH7AevsE+yKOIw=</latexit><latexit sha1_base64="DHtV0M4TMijiq/4AbB2ReXOko4=">AB7HicdVBNS8NAEJ3Ur1q/qh69LBbBU0liaOtBKHrx4KGC/YA2lM12y7dbMLuRihv8GLB0W8+oO8+W/ctBVU9MHA470ZuYFMWdK2/aHlVtZXVvfyG8WtrZ3dveK+wctFSWS0CaJeCQ7AVaUM0GbmlO7GkOAw4bQeTq8xv31OpWCTu9DSmfohHg0ZwdpITY4u0E2/WL57WK61WQXbtquM6GXGr3pmHKNkKMESjX7xvTeISBJSoQnHSnUdO9Z+iqVmhNZoZcoGmMywSPaNVTgkCo/nR87QydGaBhJE0Jjebq94kUh0pNw8B0hliP1W8vE/yuoke1vyUiTjRVJDFomHCkY5Q9jkaMEmJ5lNDMJHM3IrIGEtMtMmnYEL4+hT9T1pu2TH81ivVL5dx5OEIjuEUHKhCHa6hAU0gwOABnuDZEtaj9WK9Llpz1nLmEH7AevsE+yKOIw=</latexit><latexit sha1_base64="DHtV0M4TMijiq/4AbB2ReXOko4=">AB7HicdVBNS8NAEJ3Ur1q/qh69LBbBU0liaOtBKHrx4KGC/YA2lM12y7dbMLuRihv8GLB0W8+oO8+W/ctBVU9MHA470ZuYFMWdK2/aHlVtZXVvfyG8WtrZ3dveK+wctFSWS0CaJeCQ7AVaUM0GbmlO7GkOAw4bQeTq8xv31OpWCTu9DSmfohHg0ZwdpITY4u0E2/WL57WK61WQXbtquM6GXGr3pmHKNkKMESjX7xvTeISBJSoQnHSnUdO9Z+iqVmhNZoZcoGmMywSPaNVTgkCo/nR87QydGaBhJE0Jjebq94kUh0pNw8B0hliP1W8vE/yuoke1vyUiTjRVJDFomHCkY5Q9jkaMEmJ5lNDMJHM3IrIGEtMtMmnYEL4+hT9T1pu2TH81ivVL5dx5OEIjuEUHKhCHa6hAU0gwOABnuDZEtaj9WK9Llpz1nLmEH7AevsE+yKOIw=</latexit><latexit sha1_base64="DHtV0M4TMijiq/4AbB2ReXOko4=">AB7HicdVBNS8NAEJ3Ur1q/qh69LBbBU0liaOtBKHrx4KGC/YA2lM12y7dbMLuRihv8GLB0W8+oO8+W/ctBVU9MHA470ZuYFMWdK2/aHlVtZXVvfyG8WtrZ3dveK+wctFSWS0CaJeCQ7AVaUM0GbmlO7GkOAw4bQeTq8xv31OpWCTu9DSmfohHg0ZwdpITY4u0E2/WL57WK61WQXbtquM6GXGr3pmHKNkKMESjX7xvTeISBJSoQnHSnUdO9Z+iqVmhNZoZcoGmMywSPaNVTgkCo/nR87QydGaBhJE0Jjebq94kUh0pNw8B0hliP1W8vE/yuoke1vyUiTjRVJDFomHCkY5Q9jkaMEmJ5lNDMJHM3IrIGEtMtMmnYEL4+hT9T1pu2TH81ivVL5dx5OEIjuEUHKhCHa6hAU0gwOABnuDZEtaj9WK9Llpz1nLmEH7AevsE+yKOIw=</latexit>

<latexit sha1_base64="pu3vPW8DGU7bRS/w1IKehUeaUA=">AB6nicdZBNS8NAEIYn9avWr6pHL4tF8FQSUfRY9OKxom2FNpTNdtMu3WzC7kQoT/BiwdFvPqLvPlv3LQpqOgLCy/PzLAzb5BIYdB1P53S0vLK6lp5vbKxubW9U93da5s41Yy3WCxjfR9Qw6VQvIUCJb9PNKdRIHknGF/l9c4D10bE6g4nCfcjOlQiFIyiRbfYF/1qza2fubmIW3cXpiBeQWpQqNmvfvQGMUsjrpBJakzXcxP0M6pRMmnlV5qeELZmA51pFI278bLbqlBxZMiBhrO1TSGb0+0RGI2MmUWA7I4oj87uWw79q3RTDCz8TKkmRKzb/KEwlwZjkd5OB0JyhnFhDmRZ2V8JGVFOGNp2KDWFxKfnftE/qnvU3p7XGZRFHGQ7gEI7Bg3NowDU0oQUMhvAIz/DiSOfJeXe5q0lp5jZhx9y3r8AaQyN3g=</latexit><latexit sha1_base64="pu3vPW8DGU7bRS/w1IKehUeaUA=">AB6nicdZBNS8NAEIYn9avWr6pHL4tF8FQSUfRY9OKxom2FNpTNdtMu3WzC7kQoT/BiwdFvPqLvPlv3LQpqOgLCy/PzLAzb5BIYdB1P53S0vLK6lp5vbKxubW9U93da5s41Yy3WCxjfR9Qw6VQvIUCJb9PNKdRIHknGF/l9c4D10bE6g4nCfcjOlQiFIyiRbfYF/1qza2fubmIW3cXpiBeQWpQqNmvfvQGMUsjrpBJakzXcxP0M6pRMmnlV5qeELZmA51pFI278bLbqlBxZMiBhrO1TSGb0+0RGI2MmUWA7I4oj87uWw79q3RTDCz8TKkmRKzb/KEwlwZjkd5OB0JyhnFhDmRZ2V8JGVFOGNp2KDWFxKfnftE/qnvU3p7XGZRFHGQ7gEI7Bg3NowDU0oQUMhvAIz/DiSOfJeXe5q0lp5jZhx9y3r8AaQyN3g=</latexit><latexit sha1_base64="pu3vPW8DGU7bRS/w1IKehUeaUA=">AB6nicdZBNS8NAEIYn9avWr6pHL4tF8FQSUfRY9OKxom2FNpTNdtMu3WzC7kQoT/BiwdFvPqLvPlv3LQpqOgLCy/PzLAzb5BIYdB1P53S0vLK6lp5vbKxubW9U93da5s41Yy3WCxjfR9Qw6VQvIUCJb9PNKdRIHknGF/l9c4D10bE6g4nCfcjOlQiFIyiRbfYF/1qza2fubmIW3cXpiBeQWpQqNmvfvQGMUsjrpBJakzXcxP0M6pRMmnlV5qeELZmA51pFI278bLbqlBxZMiBhrO1TSGb0+0RGI2MmUWA7I4oj87uWw79q3RTDCz8TKkmRKzb/KEwlwZjkd5OB0JyhnFhDmRZ2V8JGVFOGNp2KDWFxKfnftE/qnvU3p7XGZRFHGQ7gEI7Bg3NowDU0oQUMhvAIz/DiSOfJeXe5q0lp5jZhx9y3r8AaQyN3g=</latexit><latexit sha1_base64="pu3vPW8DGU7bRS/w1IKehUeaUA=">AB6nicdZBNS8NAEIYn9avWr6pHL4tF8FQSUfRY9OKxom2FNpTNdtMu3WzC7kQoT/BiwdFvPqLvPlv3LQpqOgLCy/PzLAzb5BIYdB1P53S0vLK6lp5vbKxubW9U93da5s41Yy3WCxjfR9Qw6VQvIUCJb9PNKdRIHknGF/l9c4D10bE6g4nCfcjOlQiFIyiRbfYF/1qza2fubmIW3cXpiBeQWpQqNmvfvQGMUsjrpBJakzXcxP0M6pRMmnlV5qeELZmA51pFI278bLbqlBxZMiBhrO1TSGb0+0RGI2MmUWA7I4oj87uWw79q3RTDCz8TKkmRKzb/KEwlwZjkd5OB0JyhnFhDmRZ2V8JGVFOGNp2KDWFxKfnftE/qnvU3p7XGZRFHGQ7gEI7Bg3NowDU0oQUMhvAIz/DiSOfJeXe5q0lp5jZhx9y3r8AaQyN3g=</latexit>

LD(f) = X

i∈M

`(f(xi), ti))

<latexit sha1_base64="Yi2kse5MsfvkjfXhFvdabyOy4iM=">ACJ3icdVDLSsNAFJ34rPUVdelmsAgtSElF0Y1S1IULhQr2AU0Jk+mkHTqZhJmJWEL+xo2/4kZQEV36J07aVFT0wAyHc+7l3nvckFGpLOvdmJqemZ2bzy3kF5eWV1bNtfWGDCKBSR0HLBAtF0nCKCd1RUjrVAQ5LuMN3Baeo3b4iQNODXahiSjo96nHoUI6Ulxzy2faT6GLH4InHOil4JHkFbRr4TU2hTDr/sySBNmGs6BVvHVragUr/JcsWOV9KwW0ytaEZEolUwogQ80xn+xugCOfcIUZkrJdsULViZFQFDOS5O1IkhDhAeqRtqYc+UR24tGdCdzWShd6gdCPKzhSv3fEyJdy6Lu6Ml1b/vZS8S+vHSnvsBNTHkaKcDwe5EUMqgCmocEuFQrNtQEYUH1rhD3kUBY6WjzOoTJpfB/0tgtVzS/2itUT7I4cmATbIEiqIADUAXnoAbqAIM78ACewYtxbzwar8buHTKyHo2wA8YH5+iNaSD</latexit><latexit sha1_base64="Yi2kse5MsfvkjfXhFvdabyOy4iM=">ACJ3icdVDLSsNAFJ34rPUVdelmsAgtSElF0Y1S1IULhQr2AU0Jk+mkHTqZhJmJWEL+xo2/4kZQEV36J07aVFT0wAyHc+7l3nvckFGpLOvdmJqemZ2bzy3kF5eWV1bNtfWGDCKBSR0HLBAtF0nCKCd1RUjrVAQ5LuMN3Baeo3b4iQNODXahiSjo96nHoUI6Ulxzy2faT6GLH4InHOil4JHkFbRr4TU2hTDr/sySBNmGs6BVvHVragUr/JcsWOV9KwW0ytaEZEolUwogQ80xn+xugCOfcIUZkrJdsULViZFQFDOS5O1IkhDhAeqRtqYc+UR24tGdCdzWShd6gdCPKzhSv3fEyJdy6Lu6Ml1b/vZS8S+vHSnvsBNTHkaKcDwe5EUMqgCmocEuFQrNtQEYUH1rhD3kUBY6WjzOoTJpfB/0tgtVzS/2itUT7I4cmATbIEiqIADUAXnoAbqAIM78ACewYtxbzwar8buHTKyHo2wA8YH5+iNaSD</latexit><latexit sha1_base64="Yi2kse5MsfvkjfXhFvdabyOy4iM=">ACJ3icdVDLSsNAFJ34rPUVdelmsAgtSElF0Y1S1IULhQr2AU0Jk+mkHTqZhJmJWEL+xo2/4kZQEV36J07aVFT0wAyHc+7l3nvckFGpLOvdmJqemZ2bzy3kF5eWV1bNtfWGDCKBSR0HLBAtF0nCKCd1RUjrVAQ5LuMN3Baeo3b4iQNODXahiSjo96nHoUI6Ulxzy2faT6GLH4InHOil4JHkFbRr4TU2hTDr/sySBNmGs6BVvHVragUr/JcsWOV9KwW0ytaEZEolUwogQ80xn+xugCOfcIUZkrJdsULViZFQFDOS5O1IkhDhAeqRtqYc+UR24tGdCdzWShd6gdCPKzhSv3fEyJdy6Lu6Ml1b/vZS8S+vHSnvsBNTHkaKcDwe5EUMqgCmocEuFQrNtQEYUH1rhD3kUBY6WjzOoTJpfB/0tgtVzS/2itUT7I4cmATbIEiqIADUAXnoAbqAIM78ACewYtxbzwar8buHTKyHo2wA8YH5+iNaSD</latexit><latexit sha1_base64="Yi2kse5MsfvkjfXhFvdabyOy4iM=">ACJ3icdVDLSsNAFJ34rPUVdelmsAgtSElF0Y1S1IULhQr2AU0Jk+mkHTqZhJmJWEL+xo2/4kZQEV36J07aVFT0wAyHc+7l3nvckFGpLOvdmJqemZ2bzy3kF5eWV1bNtfWGDCKBSR0HLBAtF0nCKCd1RUjrVAQ5LuMN3Baeo3b4iQNODXahiSjo96nHoUI6Ulxzy2faT6GLH4InHOil4JHkFbRr4TU2hTDr/sySBNmGs6BVvHVragUr/JcsWOV9KwW0ytaEZEolUwogQ80xn+xugCOfcIUZkrJdsULViZFQFDOS5O1IkhDhAeqRtqYc+UR24tGdCdzWShd6gdCPKzhSv3fEyJdy6Lu6Ml1b/vZS8S+vHSnvsBNTHkaKcDwe5EUMqgCmocEuFQrNtQEYUH1rhD3kUBY6WjzOoTJpfB/0tgtVzS/2itUT7I4cmATbIEiqIADUAXnoAbqAIM78ACewYtxbzwar8buHTKyHo2wA8YH5+iNaSD</latexit>

f(x) = φL+1(W LφL(· · · φ0(W 0x + b0) · · · ) + bL)

<latexit sha1_base64="abQTxEG4CZzViAZPbPqe/D58jg=">AEvHictVNda9swFVjb+uyj6b417EwiAhI1iOm9gPHWV72cMeOliaQuwGWZETEX9hyaXB6E/2bf9mspPC5mwjge2C4HDvuUfnXiE/DRkXhvH9qKHpjx4/OX7afPb8xcuT1umrK57kGaFjkoRJdu1jTkMW07FgIqTXaUZx5Id04q8+lfXJLc04S+JvYp1SL8KLmAWMYKFSs9PGfdAp3Epnmi18rzD6Z0YZ742+8QA2GXknu/Ac1tiqgBCq2Gg0LNmOY5vIlm6ZDdFje3YQ9MaVuwRMqs2c2QNLPmlh6SUdSuGbQ3sknU2cJA1KOmWbdlITm72FpbyMf763ZcMk8EP3Aje+ob/2UbSrVu9y/PDXs7s/3Jgn+Ihe5mcd3ev1dXz9KdtdoP08BdgLagDbZxOWvdu/OE5BGNBQkx51NkpMIrcCYCalsujmnKSYrvKBTBWMcUe4VlSEJ36nMHAZJpk4sYJX9uaPAEefryFfMCIslr9fK5O9q01wEtlewOM0FjcnmoiAPoUhg+ZPhnGWUiHCtACYZU14hWeIME6H+e1MtAdVH3gVXZh8p/NVqX3zcruMYvAFvQcgMAIX4DO4BGNANEebaUuN6R/0ub7Sow21cbTteQ1+Cf32B1o6V48=</latexit><latexit sha1_base64="abQTxEG4CZzViAZPbPqe/D58jg=">AEvHictVNda9swFVjb+uyj6b417EwiAhI1iOm9gPHWV72cMeOliaQuwGWZETEX9hyaXB6E/2bf9mspPC5mwjge2C4HDvuUfnXiE/DRkXhvH9qKHpjx4/OX7afPb8xcuT1umrK57kGaFjkoRJdu1jTkMW07FgIqTXaUZx5Id04q8+lfXJLc04S+JvYp1SL8KLmAWMYKFSs9PGfdAp3Epnmi18rzD6Z0YZ742+8QA2GXknu/Ac1tiqgBCq2Gg0LNmOY5vIlm6ZDdFje3YQ9MaVuwRMqs2c2QNLPmlh6SUdSuGbQ3sknU2cJA1KOmWbdlITm72FpbyMf763ZcMk8EP3Aje+ob/2UbSrVu9y/PDXs7s/3Jgn+Ihe5mcd3ev1dXz9KdtdoP08BdgLagDbZxOWvdu/OE5BGNBQkx51NkpMIrcCYCalsujmnKSYrvKBTBWMcUe4VlSEJ36nMHAZJpk4sYJX9uaPAEefryFfMCIslr9fK5O9q01wEtlewOM0FjcnmoiAPoUhg+ZPhnGWUiHCtACYZU14hWeIME6H+e1MtAdVH3gVXZh8p/NVqX3zcruMYvAFvQcgMAIX4DO4BGNANEebaUuN6R/0ub7Sow21cbTteQ1+Cf32B1o6V48=</latexit><latexit sha1_base64="abQTxEG4CZzViAZPbPqe/D58jg=">AEvHictVNda9swFVjb+uyj6b417EwiAhI1iOm9gPHWV72cMeOliaQuwGWZETEX9hyaXB6E/2bf9mspPC5mwjge2C4HDvuUfnXiE/DRkXhvH9qKHpjx4/OX7afPb8xcuT1umrK57kGaFjkoRJdu1jTkMW07FgIqTXaUZx5Id04q8+lfXJLc04S+JvYp1SL8KLmAWMYKFSs9PGfdAp3Epnmi18rzD6Z0YZ742+8QA2GXknu/Ac1tiqgBCq2Gg0LNmOY5vIlm6ZDdFje3YQ9MaVuwRMqs2c2QNLPmlh6SUdSuGbQ3sknU2cJA1KOmWbdlITm72FpbyMf763ZcMk8EP3Aje+ob/2UbSrVu9y/PDXs7s/3Jgn+Ihe5mcd3ev1dXz9KdtdoP08BdgLagDbZxOWvdu/OE5BGNBQkx51NkpMIrcCYCalsujmnKSYrvKBTBWMcUe4VlSEJ36nMHAZJpk4sYJX9uaPAEefryFfMCIslr9fK5O9q01wEtlewOM0FjcnmoiAPoUhg+ZPhnGWUiHCtACYZU14hWeIME6H+e1MtAdVH3gVXZh8p/NVqX3zcruMYvAFvQcgMAIX4DO4BGNANEebaUuN6R/0ub7Sow21cbTteQ1+Cf32B1o6V48=</latexit><latexit sha1_base64="abQTxEG4CZzViAZPbPqe/D58jg=">AEvHictVNda9swFVjb+uyj6b417EwiAhI1iOm9gPHWV72cMeOliaQuwGWZETEX9hyaXB6E/2bf9mspPC5mwjge2C4HDvuUfnXiE/DRkXhvH9qKHpjx4/OX7afPb8xcuT1umrK57kGaFjkoRJdu1jTkMW07FgIqTXaUZx5Id04q8+lfXJLc04S+JvYp1SL8KLmAWMYKFSs9PGfdAp3Epnmi18rzD6Z0YZ742+8QA2GXknu/Ac1tiqgBCq2Gg0LNmOY5vIlm6ZDdFje3YQ9MaVuwRMqs2c2QNLPmlh6SUdSuGbQ3sknU2cJA1KOmWbdlITm72FpbyMf763ZcMk8EP3Aje+ob/2UbSrVu9y/PDXs7s/3Jgn+Ihe5mcd3ev1dXz9KdtdoP08BdgLagDbZxOWvdu/OE5BGNBQkx51NkpMIrcCYCalsujmnKSYrvKBTBWMcUe4VlSEJ36nMHAZJpk4sYJX9uaPAEefryFfMCIslr9fK5O9q01wEtlewOM0FjcnmoiAPoUhg+ZPhnGWUiHCtACYZU14hWeIME6H+e1MtAdVH3gVXZh8p/NVqX3zcruMYvAFvQcgMAIX4DO4BGNANEebaUuN6R/0ub7Sow21cbTteQ1+Cf32B1o6V48=</latexit>

L(f) = JD 2 LD(f) +

l=0

✓ Nl 2σ2

W l

αβW l αβ +

1 2σ2

αbl α

◆

<latexit sha1_base64="bZWGlCKpBg+bvQT3QKPjMagSsMA=">AC9XicbVJdixMxFM2MX2tX3a4+hIsQhelzLQjg/CovsgsgKdrvQTIdMmpmGzXyQZFxKyP/wxQdFfPW/+Oa/MdPpYu3uhcDh3HvOTQ5JKs6k8rw/jnvj5q3bd3budnbv3X+w191/eCrLWhA6JiUvxVmCJeWsoGPFKdnlaA4TzidJOdvm/7kMxWSlcUntaxolOsYCkjWFkq3nd2UY7VgmCuj0/PYCvoUYr26nIkh7gxdeU8+9gXcJWsagVGCi38dHRg/Nhkt8ZH0MfLZt5IXBKPQb/eiVH4waxyAMQt8gWex5na1Z2bHEHGaKtiHrf+HmFt/JFmW4/hiNjSTGY81wrxaYIgSqvA1jF3eqv1/2sRqYRK3czO+CZFg2UIdmLjbu3wmvAr8NeiBdZ3E3d9oXpI6p4UiHEs59b1KRoLxQinpoNqStMznFGpxYWOKcy0qtUDHxqmTlMS2FPoeCK3VRonEu5zBM72YQrt3sNeV1vWqs0jDQrqlrRgrSL0pDVcLmC8A5E5QovrQAE8HsXSFZYBuXsh+lY0Pwt598FZwOB7FH4Pe4Zt1HDvgMXgC+sAHL8EheAdOwBgQRzhfnG/Od/fC/er+cH+2o6z1jwC/5X76y9LEuZq</latexit><latexit sha1_base64="bZWGlCKpBg+bvQT3QKPjMagSsMA=">AC9XicbVJdixMxFM2MX2tX3a4+hIsQhelzLQjg/CovsgsgKdrvQTIdMmpmGzXyQZFxKyP/wxQdFfPW/+Oa/MdPpYu3uhcDh3HvOTQ5JKs6k8rw/jnvj5q3bd3budnbv3X+w191/eCrLWhA6JiUvxVmCJeWsoGPFKdnlaA4TzidJOdvm/7kMxWSlcUntaxolOsYCkjWFkq3nd2UY7VgmCuj0/PYCvoUYr26nIkh7gxdeU8+9gXcJWsagVGCi38dHRg/Nhkt8ZH0MfLZt5IXBKPQb/eiVH4waxyAMQt8gWex5na1Z2bHEHGaKtiHrf+HmFt/JFmW4/hiNjSTGY81wrxaYIgSqvA1jF3eqv1/2sRqYRK3czO+CZFg2UIdmLjbu3wmvAr8NeiBdZ3E3d9oXpI6p4UiHEs59b1KRoLxQinpoNqStMznFGpxYWOKcy0qtUDHxqmTlMS2FPoeCK3VRonEu5zBM72YQrt3sNeV1vWqs0jDQrqlrRgrSL0pDVcLmC8A5E5QovrQAE8HsXSFZYBuXsh+lY0Pwt598FZwOB7FH4Pe4Zt1HDvgMXgC+sAHL8EheAdOwBgQRzhfnG/Od/fC/er+cH+2o6z1jwC/5X76y9LEuZq</latexit><latexit sha1_base64="bZWGlCKpBg+bvQT3QKPjMagSsMA=">AC9XicbVJdixMxFM2MX2tX3a4+hIsQhelzLQjg/CovsgsgKdrvQTIdMmpmGzXyQZFxKyP/wxQdFfPW/+Oa/MdPpYu3uhcDh3HvOTQ5JKs6k8rw/jnvj5q3bd3budnbv3X+w191/eCrLWhA6JiUvxVmCJeWsoGPFKdnlaA4TzidJOdvm/7kMxWSlcUntaxolOsYCkjWFkq3nd2UY7VgmCuj0/PYCvoUYr26nIkh7gxdeU8+9gXcJWsagVGCi38dHRg/Nhkt8ZH0MfLZt5IXBKPQb/eiVH4waxyAMQt8gWex5na1Z2bHEHGaKtiHrf+HmFt/JFmW4/hiNjSTGY81wrxaYIgSqvA1jF3eqv1/2sRqYRK3czO+CZFg2UIdmLjbu3wmvAr8NeiBdZ3E3d9oXpI6p4UiHEs59b1KRoLxQinpoNqStMznFGpxYWOKcy0qtUDHxqmTlMS2FPoeCK3VRonEu5zBM72YQrt3sNeV1vWqs0jDQrqlrRgrSL0pDVcLmC8A5E5QovrQAE8HsXSFZYBuXsh+lY0Pwt598FZwOB7FH4Pe4Zt1HDvgMXgC+sAHL8EheAdOwBgQRzhfnG/Od/fC/er+cH+2o6z1jwC/5X76y9LEuZq</latexit><latexit sha1_base64="bZWGlCKpBg+bvQT3QKPjMagSsMA=">AC9XicbVJdixMxFM2MX2tX3a4+hIsQhelzLQjg/CovsgsgKdrvQTIdMmpmGzXyQZFxKyP/wxQdFfPW/+Oa/MdPpYu3uhcDh3HvOTQ5JKs6k8rw/jnvj5q3bd3budnbv3X+w191/eCrLWhA6JiUvxVmCJeWsoGPFKdnlaA4TzidJOdvm/7kMxWSlcUntaxolOsYCkjWFkq3nd2UY7VgmCuj0/PYCvoUYr26nIkh7gxdeU8+9gXcJWsagVGCi38dHRg/Nhkt8ZH0MfLZt5IXBKPQb/eiVH4waxyAMQt8gWex5na1Z2bHEHGaKtiHrf+HmFt/JFmW4/hiNjSTGY81wrxaYIgSqvA1jF3eqv1/2sRqYRK3czO+CZFg2UIdmLjbu3wmvAr8NeiBdZ3E3d9oXpI6p4UiHEs59b1KRoLxQinpoNqStMznFGpxYWOKcy0qtUDHxqmTlMS2FPoeCK3VRonEu5zBM72YQrt3sNeV1vWqs0jDQrqlrRgrSL0pDVcLmC8A5E5QovrQAE8HsXSFZYBuXsh+lY0Pwt598FZwOB7FH4Pe4Zt1HDvgMXgC+sAHL8EheAdOwBgQRzhfnG/Od/fC/er+cH+2o6z1jwC/5X76y9LEuZq</latexit>

data not data

SLIDE 5

What do we (usually) want from  a neural network?

Point of this paper: if you give up on all these, you can have your stat mech model. But what have we learned?

Network architecture can accommodate “infinite” data without

the network growing in size (we’re drawing from a distribution)

L2 weight/bias regularization is great, but what you really care about

is the loss function on the data

You want to train your network, not just use random weights

and biases

Loss depends on input data: you actually want the network to

learn something

SLIDE 6

Choosing a random network

Goal of the paper is to study the statistics of an ensemble of  random networks. What measure should we choose? “As the objective we hope to minimize is the total loss, with reference  to Jaynes [“Information theory and statistical mechanics”, Phys. Rev. 1957]   we select the maximum entropy distribution over  f subject to a measurement of the expected loss function.” What’s the maximum entropy distribution holding the average loss (energy)  fixed? Why, lo and behold, it’s the Boltzmann distribution:

P(f) = e−L(f)/Q

<latexit sha1_base64="zxkK1+x3vsue+Uow5Heg+CY4hLI=">ACBXicbZC7SgNBFIZn4y3G26qlFoNBiIVxVwRthKCNhUC5gJDLOTs8mQ2Qszs0JYtrHxVWwsFLH1Hex8G2eTLTxh4GP/5zDnPM7IWdSWda3kVtYXFpeya8W1tY3NrfM7Z2GDCJBoU4DHoiWQyRw5kNdMcWhFQognsOh6Yyu03rzAYRkgX+nxiF0PTLwmcsoUdrqmfvVknuELzHcx8cdj6ghJTy+TbR5Ukt6ZtEqWxPhebAzKJM1Z751ekHNPLAV5QTKdu2FapuTIRilENS6EQSQkJHZABtjT7xQHbjyRUJPtROH7uB0M9XeOL+noiJ+XYc3RnuqicraXmf7V2pNyLbsz8MFLg0+lHbsSxCnAaCe4zAVTxsQZCBdO7YjokglClgyvoEOzZk+ehcVq2NdfOipWrLI482kMHqIRsdI4q6AZVUR1R9Iie0St6M56MF+Pd+Ji25oxsZhf9kfH5A9VsluA=</latexit><latexit sha1_base64="zxkK1+x3vsue+Uow5Heg+CY4hLI=">ACBXicbZC7SgNBFIZn4y3G26qlFoNBiIVxVwRthKCNhUC5gJDLOTs8mQ2Qszs0JYtrHxVWwsFLH1Hex8G2eTLTxh4GP/5zDnPM7IWdSWda3kVtYXFpeya8W1tY3NrfM7Z2GDCJBoU4DHoiWQyRw5kNdMcWhFQognsOh6Yyu03rzAYRkgX+nxiF0PTLwmcsoUdrqmfvVknuELzHcx8cdj6ghJTy+TbR5Ukt6ZtEqWxPhebAzKJM1Z751ekHNPLAV5QTKdu2FapuTIRilENS6EQSQkJHZABtjT7xQHbjyRUJPtROH7uB0M9XeOL+noiJ+XYc3RnuqicraXmf7V2pNyLbsz8MFLg0+lHbsSxCnAaCe4zAVTxsQZCBdO7YjokglClgyvoEOzZk+ehcVq2NdfOipWrLI482kMHqIRsdI4q6AZVUR1R9Iie0St6M56MF+Pd+Ji25oxsZhf9kfH5A9VsluA=</latexit><latexit sha1_base64="zxkK1+x3vsue+Uow5Heg+CY4hLI=">ACBXicbZC7SgNBFIZn4y3G26qlFoNBiIVxVwRthKCNhUC5gJDLOTs8mQ2Qszs0JYtrHxVWwsFLH1Hex8G2eTLTxh4GP/5zDnPM7IWdSWda3kVtYXFpeya8W1tY3NrfM7Z2GDCJBoU4DHoiWQyRw5kNdMcWhFQognsOh6Yyu03rzAYRkgX+nxiF0PTLwmcsoUdrqmfvVknuELzHcx8cdj6ghJTy+TbR5Ukt6ZtEqWxPhebAzKJM1Z751ekHNPLAV5QTKdu2FapuTIRilENS6EQSQkJHZABtjT7xQHbjyRUJPtROH7uB0M9XeOL+noiJ+XYc3RnuqicraXmf7V2pNyLbsz8MFLg0+lHbsSxCnAaCe4zAVTxsQZCBdO7YjokglClgyvoEOzZk+ehcVq2NdfOipWrLI482kMHqIRsdI4q6AZVUR1R9Iie0St6M56MF+Pd+Ji25oxsZhf9kfH5A9VsluA=</latexit><latexit sha1_base64="zxkK1+x3vsue+Uow5Heg+CY4hLI=">ACBXicbZC7SgNBFIZn4y3G26qlFoNBiIVxVwRthKCNhUC5gJDLOTs8mQ2Qszs0JYtrHxVWwsFLH1Hex8G2eTLTxh4GP/5zDnPM7IWdSWda3kVtYXFpeya8W1tY3NrfM7Z2GDCJBoU4DHoiWQyRw5kNdMcWhFQognsOh6Yyu03rzAYRkgX+nxiF0PTLwmcsoUdrqmfvVknuELzHcx8cdj6ghJTy+TbR5Ukt6ZtEqWxPhebAzKJM1Z751ekHNPLAV5QTKdu2FapuTIRilENS6EQSQkJHZABtjT7xQHbjyRUJPtROH7uB0M9XeOL+noiJ+XYc3RnuqicraXmf7V2pNyLbsz8MFLg0+lHbsSxCnAaCe4zAVTxsQZCBdO7YjokglClgyvoEOzZk+ehcVq2NdfOipWrLI482kMHqIRsdI4q6AZVUR1R9Iie0St6M56MF+Pd+Ji25oxsZhf9kfH5A9VsluA=</latexit>

(Does anybody know the Jaynes reference? This seems like words  draped around the conclusion they wanted from the beginning:  if you start with the canonical ensemble, you probably get a stat mech system)

Q = Z [dW] [db] e−L

<latexit sha1_base64="xz1AZBQrtJoKdGOoLFQuOEQp/f8=">ACEnicbVDLSsNAFJ3UV62vqEs3g0VQ0JKIoBul4MaFixbsA5JYJpNJO3TyYGYilJBvcOvuHGhiFtX7vwbJ2kW2npguIdz72XOPW7MqJCG8a1VFhaXleq7W19Y3NLX17pyuihGPSwRGLeN9FgjAako6kpF+zAkKXEZ67vg67/ceCBc0Cu/kJCZOgIYh9SlGUkD/agNL6FNQwktr+dA+1hVt6jkPj2xAyRHGLH0NsGet1oGAXgPDFLUgclWgP9y/YinAQklJghISzTiKWTIi4pZiSr2YkgMcJjNCSWoiEKiHDS4qQMHijFg37E1VPmCvX3RoCISaBqyZzj2K2l4v/9axE+hdOSsM4kSTE04/8hEZwTwf6FOsGQTRDmVHmFeIQ4wlKlWFMhmLMnz5PuacNUvH1Wb16VcVTBHtgHh8AE56AJbkALdAGj+AZvI37Ul70d61j+loRSt3dsEfaJ8/1c2buQ=</latexit><latexit sha1_base64="xz1AZBQrtJoKdGOoLFQuOEQp/f8=">ACEnicbVDLSsNAFJ3UV62vqEs3g0VQ0JKIoBul4MaFixbsA5JYJpNJO3TyYGYilJBvcOvuHGhiFtX7vwbJ2kW2npguIdz72XOPW7MqJCG8a1VFhaXleq7W19Y3NLX17pyuihGPSwRGLeN9FgjAako6kpF+zAkKXEZ67vg67/ceCBc0Cu/kJCZOgIYh9SlGUkD/agNL6FNQwktr+dA+1hVt6jkPj2xAyRHGLH0NsGet1oGAXgPDFLUgclWgP9y/YinAQklJghISzTiKWTIi4pZiSr2YkgMcJjNCSWoiEKiHDS4qQMHijFg37E1VPmCvX3RoCISaBqyZzj2K2l4v/9axE+hdOSsM4kSTE04/8hEZwTwf6FOsGQTRDmVHmFeIQ4wlKlWFMhmLMnz5PuacNUvH1Wb16VcVTBHtgHh8AE56AJbkALdAGj+AZvI37Ul70d61j+loRSt3dsEfaJ8/1c2buQ=</latexit><latexit sha1_base64="xz1AZBQrtJoKdGOoLFQuOEQp/f8=">ACEnicbVDLSsNAFJ3UV62vqEs3g0VQ0JKIoBul4MaFixbsA5JYJpNJO3TyYGYilJBvcOvuHGhiFtX7vwbJ2kW2npguIdz72XOPW7MqJCG8a1VFhaXleq7W19Y3NLX17pyuihGPSwRGLeN9FgjAako6kpF+zAkKXEZ67vg67/ceCBc0Cu/kJCZOgIYh9SlGUkD/agNL6FNQwktr+dA+1hVt6jkPj2xAyRHGLH0NsGet1oGAXgPDFLUgclWgP9y/YinAQklJghISzTiKWTIi4pZiSr2YkgMcJjNCSWoiEKiHDS4qQMHijFg37E1VPmCvX3RoCISaBqyZzj2K2l4v/9axE+hdOSsM4kSTE04/8hEZwTwf6FOsGQTRDmVHmFeIQ4wlKlWFMhmLMnz5PuacNUvH1Wb16VcVTBHtgHh8AE56AJbkALdAGj+AZvI37Ul70d61j+loRSt3dsEfaJ8/1c2buQ=</latexit><latexit sha1_base64="xz1AZBQrtJoKdGOoLFQuOEQp/f8=">ACEnicbVDLSsNAFJ3UV62vqEs3g0VQ0JKIoBul4MaFixbsA5JYJpNJO3TyYGYilJBvcOvuHGhiFtX7vwbJ2kW2npguIdz72XOPW7MqJCG8a1VFhaXleq7W19Y3NLX17pyuihGPSwRGLeN9FgjAako6kpF+zAkKXEZ67vg67/ceCBc0Cu/kJCZOgIYh9SlGUkD/agNL6FNQwktr+dA+1hVt6jkPj2xAyRHGLH0NsGet1oGAXgPDFLUgclWgP9y/YinAQklJghISzTiKWTIi4pZiSr2YkgMcJjNCSWoiEKiHDS4qQMHijFg37E1VPmCvX3RoCISaBqyZzj2K2l4v/9axE+hdOSsM4kSTE04/8hEZwTwf6FOsGQTRDmVHmFeIQ4wlKlWFMhmLMnz5PuacNUvH1Wb16VcVTBHtgHh8AE56AJbkALdAGj+AZvI37Ul70d61j+loRSt3dsEfaJ8/1c2buQ=</latexit>

SLIDE 7

Changing variables

The nontrivial result of the paper is that pre-activations  are a nicer choice of variables than weights and biases. …at the price of coupling  all the pre-activations  together.

What is the temperature? What else can we compute from Q?
Can we do perturbation theory in JD?
Can we treat the input data as an external source field?

Things which are not addressed in this paper but may be worth investigating:

Data loss only depends 

n the pre-activation 
f the final layer…

Q = Z [dz][d] exp " −JD 2 X

i∈M

`(L+1(zL

i ), ti)− L

l=0

⇢1 2(λl

α)T Σlλl α − i(λl α)T zl α

# . (37)

Σl = σ2

wN −1 l

φl

α(φl α)T + 1σ2 b

<latexit sha1_base64="dlrTZLfNuPfp64WVwvNXrYF+C8g=">ACWnicbZFNSxtBGMefrPUl8S3W3noZFCEiht1c9GIJFIqnYqmJQja7PDOZTQZndpeZWSUs+Xx+hkIP4t2rvXaSeGgTHxj48X9e+Q/NpTDW939VvJUPq2vrG9Xa5tb2zm5972PXZIVmvMymelbioZLkfKOFVby21xzVFTyG3r3dZq/uefaiCy9tuOc9xUOU5EIhtZJcR3LkCYk/CmGCieRJBckNFOH6IW+R7LqDwNJiFVZiPhCuIQ5T5CEljWTuOrskJmc1zLfMpNGrF9UO/6c+CLEPwBoftg+TbEwBcxfXncJCxQvHUMonG9AI/t/0StRVM8ktLAzPkd3hkPcpqi46ZczKybkyCkDkmTavdSmfpvR4nKmLGirlKhHZnF3FR8N0fVwmabnPdLkeaF5SmbL04KSWxGpj6TgdCcWTl2gEwLdzthI9TIrPuNmjMlWLRgGbqtZuD4h3PnC8xjAz7DATQgDNowyVcQcYPMILvMKfym/P86re5rzUq7z17MN/4X36C9CUt+Q=</latexit><latexit sha1_base64="4ICY0FRKk0Jh236t0MoXteTLoPA=">ACWnicbZHJSwMxFMbTcW9d6nLzEiqCIspML3pRBE8iaLVgtMZXtJMG0xmhiSjlKF/n970Knjy7lWvpstBWx8Efnxv5QtJBdfGd8KzsTk1PTM7FyxNL+wuFReXrnRSaYoq9FEJKpOQDPBY1Yz3AhWTxUDSQS7JfcnvfztA1OaJ/G16aSsIaEV84hTMFYKy5D7JML+FW9J6AYCH2Jf9zh8DKr4PBRBvut1fSJzP21zWxD6INI24K1xbTu4xju4P8+2DKaQoBqWN9w9tx94HLwhbBxXotOX5+3Xi7D84TcTmkWGypA6zvPTU0jB2U4Faxb9DPNUqD30GJ3FmOQTDfyvhVdvGmVJo4SZV9scF/93ZGD1Lojia2UYNp6NcT/80RObLZRAeNnMdpZlhMB4ujTGCT4J7PuMkVo0Z0LABV3N6OaRsUGN/o2hN8UYtGIeb6p5n+dK6c4QGMYvWUQVtIQ/to2N0hi5QDVH0hD7RF/ouvDuOM+eUBqVOYdiziv6Es/YDLe5lA=</latexit><latexit sha1_base64="4ICY0FRKk0Jh236t0MoXteTLoPA=">ACWnicbZHJSwMxFMbTcW9d6nLzEiqCIspML3pRBE8iaLVgtMZXtJMG0xmhiSjlKF/n970Knjy7lWvpstBWx8Efnxv5QtJBdfGd8KzsTk1PTM7FyxNL+wuFReXrnRSaYoq9FEJKpOQDPBY1Yz3AhWTxUDSQS7JfcnvfztA1OaJ/G16aSsIaEV84hTMFYKy5D7JML+FW9J6AYCH2Jf9zh8DKr4PBRBvut1fSJzP21zWxD6INI24K1xbTu4xju4P8+2DKaQoBqWN9w9tx94HLwhbBxXotOX5+3Xi7D84TcTmkWGypA6zvPTU0jB2U4Faxb9DPNUqD30GJ3FmOQTDfyvhVdvGmVJo4SZV9scF/93ZGD1Lojia2UYNp6NcT/80RObLZRAeNnMdpZlhMB4ujTGCT4J7PuMkVo0Z0LABV3N6OaRsUGN/o2hN8UYtGIeb6p5n+dK6c4QGMYvWUQVtIQ/to2N0hi5QDVH0hD7RF/ouvDuOM+eUBqVOYdiziv6Es/YDLe5lA=</latexit><latexit sha1_base64="M7KEpAR63R2jzb0zb/oICtbMuo=">ACWnicbZHLSgMxFIbT8V5v9bJzEyxCRZSZbnSjCG5cScW2Fjqd4STNtMFkZkgyShn6fD6D4Mq9W92aXhbaeiDw8Z8rf0gquDau+1ZwFhaXldW14rG5tb26Wd3aZOMkVZgyYiUS0Cmgkes4bhRrBWqhItgjeboZ5R+fmdI8ietmkLKOhF7MI07BWCksQe6TCPsPvCdhGAh8iX094vAlqOK7UAT5qTf0icz9tM9tQeiDSPuAK/PacVDHJ3g8z7ZMpCgGpbK7pk7DjwP3hTKaBq1sPThdxOaSRYbKkDrtuempODMpwKNiz6mWYp0CfosbFGCTnXxsxRAfWaWLo0TZFxs8Vn935C1HkhiKyWYvp7NjcR/c0TObDbRSfncZoZFtPJ4igT2CR45DPucsWoEQMLQBW3t2PaBwXU2N8oWlO8WQvmoVk98yzfu+Xrq6k9q+gAHaIK8tA5uka3qIYaiKJX9Im+0Hfh3XGcNWd9UuoUpj176E84+z8XTrX2</latexit>

to elucidate its structure. To do this ), (zl

α)T = (zl α;1, zl α;2, · · · zl α;|M|),

. Finally we define the matrix

φl

α = φl(zl−1 α

)

<latexit sha1_base64="+VGs/A9PUwBbmUkvb4219Fxvyk=">ACKXicbZDLSgMxFIbP1Fut6pLQYJFqAvLjBvdKAU3LivYC3TaknTNjSZGZKMUIe+htCde1/Bre7dqUt9ETNtEW09EPj4/3M4J78Xcqa0b9bqYXFpeWV9GpmbX1jcyu7vVNRQSQJLZOAB7LmYU582lZM81pLZQUC4/Tqte/TPzqLZWKBf6NHoS0IXDXZx1GsDZSK2u7nojdsMeGLRfzsIebHJ2jRGjyfOLd/RgxP3aGR61szi7Y40Lz4EwhV9wfjR4AoNTKfrtgESC+pwrFTdsUPdiLHUjHA6zLiRoiEmfdyldYM+FlQ14vHPhujQKG3UCaR5vkZj9fdEjIVSA+GZToF1T816ifiv54mZzbpz1oiZH0a+mSyuBNxpAOUxIbaTFKi+cAJpKZ2xHpYmJNuFmTCjObATzUDkpOIavToXMKk07MEB5MGBUyjCFZSgDATu4Qme4cV6tF6tN+tj0pqypjO78Kesr2/bJKlX</latexit><latexit sha1_base64="yS/HPLWYkOQCgPOaEbiroYvL6Y=">ACKXicbZBNS8MwHMbT+TbnW9WjIMEhzIOj9aIXZeDF4wT3AmtX0izdwpK2JKkwS7+GoF/Br+BV797Uo34R02ITh8I/Hief/gnjx8zKpVlvRmFufmFxaXicmldW19w9zcasoEZg0cMQi0faRJIyGpKGoYqQdC4K4z0jLH57neuaCEmj8EqNYuJy1A9pQDFS2vJMy/F56sQDmnkOYvEAdRk8hbnRZU8u/kOUnZoZweWbaq1ljwL9hTKNd273Ld1z3zw+lFOEkVJghKTu2FSs3RUJRzEhWchJYoSHqE86GkPEiXT8c8yuK+dHgwioU+o4Nj9eSNFXMoR9/UkR2ogZ7Pc/Dfz+cxmFZy4KQ3jRJEQTxYHCYMqgnltsEcFwYqNCAsqH47xAMkEFa63JIuxZ6t4C80j6q25kvdzhmYqAh2wB6oABscgxq4AHXQABjcgkfwBJ6NB+PFeDXeJ6MFY3pnG/yS8fkFVMCrHA=</latexit><latexit sha1_base64="yS/HPLWYkOQCgPOaEbiroYvL6Y=">ACKXicbZBNS8MwHMbT+TbnW9WjIMEhzIOj9aIXZeDF4wT3AmtX0izdwpK2JKkwS7+GoF/Br+BV797Uo34R02ITh8I/Hief/gnjx8zKpVlvRmFufmFxaXicmldW19w9zcasoEZg0cMQi0faRJIyGpKGoYqQdC4K4z0jLH57neuaCEmj8EqNYuJy1A9pQDFS2vJMy/F56sQDmnkOYvEAdRk8hbnRZU8u/kOUnZoZweWbaq1ljwL9hTKNd273Ld1z3zw+lFOEkVJghKTu2FSs3RUJRzEhWchJYoSHqE86GkPEiXT8c8yuK+dHgwioU+o4Nj9eSNFXMoR9/UkR2ogZ7Pc/Dfz+cxmFZy4KQ3jRJEQTxYHCYMqgnltsEcFwYqNCAsqH47xAMkEFa63JIuxZ6t4C80j6q25kvdzhmYqAh2wB6oABscgxq4AHXQABjcgkfwBJ6NB+PFeDXeJ6MFY3pnG/yS8fkFVMCrHA=</latexit><latexit sha1_base64="ql2HV7+7B+uonAkbaXsbyZQGsCs=">ACKXicbZDLSsNAFIYn9VbrLerSzWAR6sKSuNGNUnDjsoK9QJOGyXTSDJ1JwsxEqKGv4Uv4Cm51705d6os4aYNo64GBj/8/h3Pm9xNGpbKsd6O0tLyulZer2xsbm3vmLt7bRmnApMWjlksuj6ShNGItBRVjHQTQRD3Gen4o6vc79wRIWkc3apxQlyOhENKEZKS5pOT7PnCSkE89BLAlRn8ELmAt9Vsu9+x8jYyf25Ngzq1bdmhZcBLuAKiq6ZmfziDGKSeRwgxJ2bOtRLkZEopiRiYVJ5UkQXiEhqSnMUKcSDeb/mwCj7QygEs9IsUnKq/JzLEpRxzX3dypEI57+Xiv57P5zar4NzNaJSkikR4tjhIGVQxzGODAyoIVmysAWFB9e0Qh0grHS4FR2KPR/BIrRP67bmG6vauCziKYMDcAhqwAZnoAGuQRO0AYP4Ak8gxfj0Xg13oyPWvJKGb2wZ8yvr4BxFKnJw=</latexit>

SLIDE 8

Quick overview of the proof

Remember Faddeev-Popov? Introduce pre-activations  as delta-function constraints, exponentiate them to put them in the action, complete the square to integrate out weights and biases

l=0

✓ ◆ = Z [dW][db] Y

i∈M

exp " −JD 2 X

`(L(· · · 1(W 0xi + b0) · · · ), ti) #! × exp " −

l=0

✓ Nl 22

W l

αβW l αβ +

1 22

αbl α

◆# (27) = Z [dW][db] Y

i∈M

Z [dz0

i ] exp

" −JD 2 X

`(L(· · · 2(W 11(z0

i ) + b1) · · · ), ti)

# × (z0

i − W 0xi − b0)

! exp " −

l=0

✓ Nl 22

W l

αβW l αβ +

1 22

αbl α

◆# . (28)

<latexit sha1_base64="s9cpMgegKO+rBlXOoPMp9jDWQ70=">AB+HicbZA9SwNBEIbn/IzxK2qpxWIQrMKdjZBG8scmA9IjrC3mUuW7N4du3tCPILbLW3E1sr/4qd+EvcfBSa+MLAwzszPCGqeDauO6ns7K6tr6xWdgqbu/s7u2XDg4bOskUwzpLRKJaIdUoeIx1w43AVqQylBgMxzeTPrNe1SaJ/GdGaUYSNqPecQZNdby/W6p7FbcqcgyeHMoV08+/C8AqHVL351ewjKJsWGCat323NQEOVWGM4HjYifTmFI2pH1sW4ypRB3k0fH5Mw6PRIlylZsyNT9vZFTqfVIhnZSUjPQi72J+W8vlAuXTXQV5DxOM4Mxmx2OMkFMQiYpkB5XyIwYWaBMcfs7YQOqKDM2q6INxVuMYBkaFxXPsm/TuYaZCnAMp3AOHlxCFW6hBnVgPAIT/DsPDgvzqvzNhtdceY7R/BHzvsPE/CVvA=</latexit><latexit sha1_base64="1smzlzO9BaOZiD/nI4I+GagPIbY=">AB+HicbZC7SwNBEMbn4ivGV9RSkcUgWIU7Gy2DNpY5MA9IjrC32UuW7O4du3tCPFJa2WpvJ7ZW+Vdsbf0n3DwKTfxg4Mc3M8zwhQln2rjup5NbWV1b38hvFra2d3b3ivsHdR2nitAaiXmsmiHWlDNJa4YZTpuJoliEnDbCwc2k37inSrNY3plhQgOBe5JFjGBjLd/vFEtu2Z0KLYM3h1LleOx/P56Mq53iV7sbk1RQaQjHWrc8NzFBhpVhNRoZ1qmAywD3asixoDrIpo+O0Jl1uiKlS1p0NT9vZFhofVQhHZSYNPXi72J+W8vFAuXTXQVZEwmqaGSzA5HKUcmRpMUJcpSgwfWsBEMfs7In2sMDE2q4INxVuMYBnqF2XPsm/TuYaZ8nAEp3AOHlxCBW6hCjUgQOEJnuHFeXBenTfnfTac+Y7h/BHzscPCJWXIg=</latexit><latexit sha1_base64="1smzlzO9BaOZiD/nI4I+GagPIbY=">AB+HicbZC7SwNBEMbn4ivGV9RSkcUgWIU7Gy2DNpY5MA9IjrC32UuW7O4du3tCPFJa2WpvJ7ZW+Vdsbf0n3DwKTfxg4Mc3M8zwhQln2rjup5NbWV1b38hvFra2d3b3ivsHdR2nitAaiXmsmiHWlDNJa4YZTpuJoliEnDbCwc2k37inSrNY3plhQgOBe5JFjGBjLd/vFEtu2Z0KLYM3h1LleOx/P56Mq53iV7sbk1RQaQjHWrc8NzFBhpVhNRoZ1qmAywD3asixoDrIpo+O0Jl1uiKlS1p0NT9vZFhofVQhHZSYNPXi72J+W8vFAuXTXQVZEwmqaGSzA5HKUcmRpMUJcpSgwfWsBEMfs7In2sMDE2q4INxVuMYBnqF2XPsm/TuYaZ8nAEp3AOHlxCBW6hCjUgQOEJnuHFeXBenTfnfTac+Y7h/BHzscPCJWXIg=</latexit><latexit sha1_base64="H1z8dDRjWDX4QOngz/J5q2dzs=">AB+HicbZA9SwNBEIbn4leMX1FLm8UgWIU7Gy2DNpYJmA9IjrC3mUuW7O4du3tCDPkFtrbia3/xtZf4ia5QhNfGHh4Z4YZ3igV3Fjf/IKG5tb2zvF3dLe/sHhUfn4pGWSTDNskQkuhNRg4IrbFpuBXZSjVRGAtvR+G7ebz+iNjxRD3aSYijpUPGYM2qd1Wj0yxW/6i9E1iHIoQK56v3yd2+QsEyiskxQY7qBn9pwSrXlTOCs1MsMpSN6RC7DhWVaMLp4tEZuXDOgMSJdqUsWbi/N6ZUGjORkZuU1I7Mam9u/tuL5MplG9+EU67SzKJiy8NxJohNyDwFMuAamRUTB5Rp7n4nbEQ1ZdZlVXKhBKsRrEPrqho4bviV2m0eTxHO4BwuIYBrqME91KEJDBCe4QVevSfvzXv3PpajBS/fOYU/8j5/AN+Xk3c=</latexit>

{

Iterate to expand the activation function composition in the data loss:

Q = Z [dW][db] Y

i∈M L

l=0

Z [dzl

i] exp

" −JD 2 X

`(L+1(zL

i ), ti)

# ×

l=0

(zl

i − W ll(zl−1 i

) + bl) ! exp " −

l=0

✓ Nl 22

W l

αβW l αβ +

1 22

αbl α

◆# (29)

nly final layer

pre-activation all other  activations  hiding here

SLIDE 9

Quick overview of the proof

Remember Faddeev-Popov? Introduce pre-activations  as delta-function constraints, exponentiate them to put them in the action, complete the square to integrate out weights and biases

δ(x) = Z dλ e−ixλ

<latexit sha1_base64="mkoFQYQ6YO5BsxKNz6e3fmnyKk=">ACJnicbVDLSgNBEOz1bXxFPXoZDEIE3YF0YsY9OJRwUQhu4bZ2d5kcPbBzKwYlnyFP+EveNW7NxFv4pc4eRw0sWCgqOqie8pPBVfatj+tqemZ2bn5hcXC0vLK6lpxfaOhkwyrLNEJPLGpwoFj7GuRZ4k0qkS/w2r876/vX9ygVT+Ir3U3Ri2g75iFnVBupVay4AQpNyw+75Ji4PNYkcIWJB5S4ewRv8wonD2Qk9VrFkl21ByCTxBmRUu2kelADgItW8dsNEpZFGsmqFJNx061l1OpORPYK7iZwpSyO9rGpqExjVB5+eBbPbJjlICEiTPHDZQfydyGinVjXwzGVHdUeNeX/zX86OxzTo8nIep5nGmA0Xh5kgOiH9zkjAJTItuoZQJrm5nbAOlZRp02zBlOKMVzBJGvtVx/BL084pDLEAW7ANZXDgEGpwDhdQBwaP8Awv8Go9W/Wu/UxHJ2yRplN+APr6wcDjKVT</latexit><latexit sha1_base64="51dY71L/bqxhHD2csh6un6fHYXM=">ACJnicbVDLSgMxFM34rPU16tJNsAgVbJkRDfSohuXFewDOrVkMrdtaOZBkpGWsR8h/oS/4Fb37kTciXv/wfSx0NYDgcM593BvjhtxJpVlfRhz8wuLS8uplfTq2vrGprm1XZFhLCiUachDUXOJBM4CKCumONQiAcR3OVTd7sXQr96CkCwMrlU/goZP2gFrMUqUlpmzvGAK5LtHeAz7LBAYc/hOu4R7BxiuElyDPfwRBo0zYyVt0bAs8SekEyxkD8u3H3fl5rml+OFNPYhUJQTKeu2FalGQoRilMg7cQSIkK7pA1TQPig2wko28N8L5WPNwKhX76sJH6O5EQX8q+7+pJn6iOnPaG4r+e609tVq3TRsKCKFYQ0PHiVsyxCvGwM+wxAVTxviaECqZvx7RDBKFKN5vWpdjTFcySylHe1vxKt3OxkihXbSHshGJ6iILlEJlRFD+gJPaMX49F4Nd6M9/HonDHJ7KA/MD5/AMXJp0w=</latexit><latexit sha1_base64="51dY71L/bqxhHD2csh6un6fHYXM=">ACJnicbVDLSgMxFM34rPU16tJNsAgVbJkRDfSohuXFewDOrVkMrdtaOZBkpGWsR8h/oS/4Fb37kTciXv/wfSx0NYDgcM593BvjhtxJpVlfRhz8wuLS8uplfTq2vrGprm1XZFhLCiUachDUXOJBM4CKCumONQiAcR3OVTd7sXQr96CkCwMrlU/goZP2gFrMUqUlpmzvGAK5LtHeAz7LBAYc/hOu4R7BxiuElyDPfwRBo0zYyVt0bAs8SekEyxkD8u3H3fl5rml+OFNPYhUJQTKeu2FalGQoRilMg7cQSIkK7pA1TQPig2wko28N8L5WPNwKhX76sJH6O5EQX8q+7+pJn6iOnPaG4r+e609tVq3TRsKCKFYQ0PHiVsyxCvGwM+wxAVTxviaECqZvx7RDBKFKN5vWpdjTFcySylHe1vxKt3OxkihXbSHshGJ6iILlEJlRFD+gJPaMX49F4Nd6M9/HonDHJ7KA/MD5/AMXJp0w=</latexit><latexit sha1_base64="kcx1QrFUJdTgJ9QAKDjlHifKWk=">ACJnicbVDLSsNAFJ34rPUVdelmsAgVbEnc6EYounFZwT6giWUyuWmHTh7MTKQl9Cv8CX/Bre7dibgTv8Rpm4W2Hhg4nHMP987xEs6ksqxPY2l5ZXVtvbBR3Nza3tk19/abMk4FhQaNeSzaHpHAWQNxRSHdiKAhB6Hlje4nvitBxCSxdGdGiXghqQXsYBRorTUNSuOD1yR8vAEX2KHRQr7Dtdxn2DnFMN9VmF4iHNp3DVLVtWaAi8SOyclKPeNb8dP6ZpCJGinEjZsa1EuRkRilEO46KTSkgIHZAedDSNSAjSzabfGuNjrfg4iIV+rCp+juRkVDKUejpyZCovpz3JuK/nhfObVbBhZuxKEkVRHS2OEg5VjGedIZ9JoAqPtKEUMH07Zj2iSBU6WaLuhR7voJF0jyr2prfWqXaV5PAR2iI1RGNjpHNXSD6qiBKHpEz+gFvRpPxpvxbnzMRpeMPHOA/sD4+gFpUKQx</latexit>

= ⇒

<latexit sha1_base64="IPUJymiDdC3ftWqdrRuo7Hi1ME=">AB/3icbVC7SgNBFL3rM6vqKXNYhCswq6NmLQxjKCeUCyhNnJ3WTIzOwyMyuEJaC/YKu9ndj6BX6DrV/i5Fo4oELh3Pu5VxOlHKmje9/OUvLK6tr64UNd3Nre2e3uLdf10mKNZowhPVjIhGziTWDMcm6lCIiKOjWhwPfYb96g0S+SdGaYCtKTLGaUGCs120zYFNSdYskv+xN4iySYkdLlp3vxADVTvG73U1oJlAayonWrcBPTZgTZRjlOHLbmcaU0AHpYctSQTqMJ/8O/KOrdL14kTZkcabqL8vciK0HorIbgpi+nreG4v/epGYSzbxeZgzmWYGJZ0Gxn3TOKNy/C6TCE1fGgJoYrZ3z3aJ4pQYytzbSnBfAWLpH5aDiy/9UuVK5iAIdwBCcQwBlU4AaqUAMKHJ7gGV6cR+fVeXPep6tLzuzmAP7A+fgBMwyYiA=</latexit><latexit sha1_base64="xByvHhQZEp4QMd8bpHhbE4ZaKR8=">AB/3icbVC7SgNBFL0bX3F9RS1tBoNgFXZtBGDNpYRzAOSEGYnd5MhM7vLzKwQlhT+gq12FnZi6xf4CWLrlzh5FJp4MLhnHs5lxMkgmvjeV9Obml5ZXUtv+5ubG5t7xR292o6ThXDKotFrBoB1Sh4hFXDjcBGopDKQGA9GFyN/fodKs3j6NYME2xL2ot4yBk1Vmq0uLQpqDuFolfyJiCLxJ+R4sWHe548f7qVTuG71Y1ZKjEyTFCtm76XmHZGleFM4MhtpRoTyga0h01LIypRt7PJvyNyZJUuCWNlJzJkov6+yKjUeigDuymp6et5byz+6wVyLtmEZ+2MR0lqMGLT4DAVxMRkXAbpcoXMiKElClufyesTxVlxlbm2lL8+QoWSe2k5Ft+4xXLlzBFHg7gEI7Bh1MowzVUoAoMBDzAIzw5986L8+q8TVdzuxmH/7Aef8BO9uZ/A=</latexit><latexit sha1_base64="xByvHhQZEp4QMd8bpHhbE4ZaKR8=">AB/3icbVC7SgNBFL0bX3F9RS1tBoNgFXZtBGDNpYRzAOSEGYnd5MhM7vLzKwQlhT+gq12FnZi6xf4CWLrlzh5FJp4MLhnHs5lxMkgmvjeV9Obml5ZXUtv+5ubG5t7xR292o6ThXDKotFrBoB1Sh4hFXDjcBGopDKQGA9GFyN/fodKs3j6NYME2xL2ot4yBk1Vmq0uLQpqDuFolfyJiCLxJ+R4sWHe548f7qVTuG71Y1ZKjEyTFCtm76XmHZGleFM4MhtpRoTyga0h01LIypRt7PJvyNyZJUuCWNlJzJkov6+yKjUeigDuymp6et5byz+6wVyLtmEZ+2MR0lqMGLT4DAVxMRkXAbpcoXMiKElClufyesTxVlxlbm2lL8+QoWSe2k5Ft+4xXLlzBFHg7gEI7Bh1MowzVUoAoMBDzAIzw5986L8+q8TVdzuxmH/7Aef8BO9uZ/A=</latexit><latexit sha1_base64="mZema8jf7u064ZCK2Ci6ER+dKsQ=">AB/3icbVC7SgNBFL3rM8ZX1NJmMAhWYdGy6CNZQTzgGQJs5O7yZCZ2WVmVghLCn/BVns7sfVTbP0SJ8kWmnjgwuGcezmXE6WCG+v7X97a+sbm1nZp7y7t39wWDk6bpk0wybLBGJ7kTUoOAKm5ZbgZ1UI5WRwHY0vp357UfUhifqwU5SDCUdKh5zRq2TOj0uXQqafqXq1/w5yCoJClKFAo1+5bs3SFgmUVkmqDHdwE9tmFNtORM4LfcygylYzrErqOKSjRhPv93Ss6dMiBxot0oS+bq74ucSmMmMnKbktqRWfZm4r9eJeSbXwd5lylmUXFsFxJohNyKwMuAamRUTRyjT3P1O2IhqyqyrOxKCZYrWCWty1rg+L1frd8U9ZTgFM7gAgK4gjrcQOawEDAM7zAq/fkvXnv3sdidc0rbk7gD7zPH6b1lrs=</latexit>

Q = Z [dΩ] exp " − X

i∈M

( JD 2 `(L+1(zL

i ), ti) + i0 α;i(z0 α;i − W 0 αβxβ;i − b0 α;i)

l=1

α;i(zl α;i − W l αβl(zl−1 β;i ) − bl α)

) −

l=0

Nl 22

W l

αβW l αβ +

1 22

αbl α

!# (25) where we have let [dΩ] = [dW][db][dz][d] for notational convenience.

SLIDE 10

Quick overview of the proof

Remember Faddeev-Popov? Introduce pre-activations  as delta-function constraints, exponentiate them to put them in the action, complete the square to integrate out weights and biases

δ(x) = Z dλ e−ixλ

= ⇒

Q = Z [dΩ] exp " − X

i∈M

( JD 2 `(L+1(zL

i ), ti) + i0 α;i(z0 α;i − W 0 αβxβ;i − b0 α;i)

l=1

α;i(zl α;i − W l αβl(zl−1 β;i ) − bl α)

) −

l=0

Nl 22

W l

αβW l αβ +

1 22

αbl α

!# (25) where we have let [dΩ] = [dW][db][dz][d] for notational convenience.

At this stage, all weights and biases appear at most quadratically,  at the expense of adding pre-activations z and auxiliary fields to the action. Complete the square, integrate out W and b to get

Q = Z [dz][d] exp " −JD 2 X

i∈M

`(L+1(zL

i ), ti)− L

l=0

⇢1 2(λl

α)T Σlλl α − i(λl α)T zl α

# . (37)

SLIDE 11

The rest of the paper

Network architecture can accommodate “infinite” data without

the network growing in size (we’re drawing from a distribution)  require

L2 weight/bias regularization is great, but what you really care about

is the loss function on the data 

You want to train your network, not just use random weights

and biases  no training, only initialization according to Q

Loss depends on input data: you actually want the network to

learn something  put the network on a circle: Nl |M|

<latexit sha1_base64="Q2mwhPDlsAy9iydwVX/0ZGlpZKY=">AB/nicdVDLSsNAFJ3UV62vqLhyM1gEVyWpoa0bKbhxo1SwD2hCmEwn6dDJg5mJUNKCv+LGhSJu/Q53/o2TtoKHhg4nHMv98zxEkaFNIwPrbC0vLK6VlwvbWxube/ou3sdEackzaOWcx7HhKE0Yi0JZWM9BJOUOgx0vVGF7nfvSNc0Di6leOEOCEKIupTjKSXP3g2mXQDgI4sUMkhxix7Go6cfWyUTlr1KpWDRoVw6ibVTMn1bp1akFTKTnKYIGWq7/bgxinIYkZkiIvmk0skQlxQzMi3ZqSAJwiMUkL6iEQqJcLJZ/Ck8VsoA+jFXL5Jwpn7fyFAoxDj01GSeUfz2cvEvr59Kv+FkNEpSI8P+SnDMoY5l3AeUESzZWBGFOVaIh4gjLFVjJVXC10/h/6RTrZiK31jl5vmijiI4BEfgBJigDprgErRAG2CQgQfwBJ61e+1Re9Fe56MFbGzD35Ae/sERveVsQ=</latexit><latexit sha1_base64="Q2mwhPDlsAy9iydwVX/0ZGlpZKY=">AB/nicdVDLSsNAFJ3UV62vqLhyM1gEVyWpoa0bKbhxo1SwD2hCmEwn6dDJg5mJUNKCv+LGhSJu/Q53/o2TtoKHhg4nHMv98zxEkaFNIwPrbC0vLK6VlwvbWxube/ou3sdEackzaOWcx7HhKE0Yi0JZWM9BJOUOgx0vVGF7nfvSNc0Di6leOEOCEKIupTjKSXP3g2mXQDgI4sUMkhxix7Go6cfWyUTlr1KpWDRoVw6ibVTMn1bp1akFTKTnKYIGWq7/bgxinIYkZkiIvmk0skQlxQzMi3ZqSAJwiMUkL6iEQqJcLJZ/Ck8VsoA+jFXL5Jwpn7fyFAoxDj01GSeUfz2cvEvr59Kv+FkNEpSI8P+SnDMoY5l3AeUESzZWBGFOVaIh4gjLFVjJVXC10/h/6RTrZiK31jl5vmijiI4BEfgBJigDprgErRAG2CQgQfwBJ61e+1Re9Fe56MFbGzD35Ae/sERveVsQ=</latexit><latexit sha1_base64="Q2mwhPDlsAy9iydwVX/0ZGlpZKY=">AB/nicdVDLSsNAFJ3UV62vqLhyM1gEVyWpoa0bKbhxo1SwD2hCmEwn6dDJg5mJUNKCv+LGhSJu/Q53/o2TtoKHhg4nHMv98zxEkaFNIwPrbC0vLK6VlwvbWxube/ou3sdEackzaOWcx7HhKE0Yi0JZWM9BJOUOgx0vVGF7nfvSNc0Di6leOEOCEKIupTjKSXP3g2mXQDgI4sUMkhxix7Go6cfWyUTlr1KpWDRoVw6ibVTMn1bp1akFTKTnKYIGWq7/bgxinIYkZkiIvmk0skQlxQzMi3ZqSAJwiMUkL6iEQqJcLJZ/Ck8VsoA+jFXL5Jwpn7fyFAoxDj01GSeUfz2cvEvr59Kv+FkNEpSI8P+SnDMoY5l3AeUESzZWBGFOVaIh4gjLFVjJVXC10/h/6RTrZiK31jl5vmijiI4BEfgBJigDprgErRAG2CQgQfwBJ61e+1Re9Fe56MFbGzD35Ae/sERveVsQ=</latexit><latexit sha1_base64="Q2mwhPDlsAy9iydwVX/0ZGlpZKY=">AB/nicdVDLSsNAFJ3UV62vqLhyM1gEVyWpoa0bKbhxo1SwD2hCmEwn6dDJg5mJUNKCv+LGhSJu/Q53/o2TtoKHhg4nHMv98zxEkaFNIwPrbC0vLK6VlwvbWxube/ou3sdEackzaOWcx7HhKE0Yi0JZWM9BJOUOgx0vVGF7nfvSNc0Di6leOEOCEKIupTjKSXP3g2mXQDgI4sUMkhxix7Go6cfWyUTlr1KpWDRoVw6ibVTMn1bp1akFTKTnKYIGWq7/bgxinIYkZkiIvmk0skQlxQzMi3ZqSAJwiMUkL6iEQqJcLJZ/Ck8VsoA+jFXL5Jwpn7fyFAoxDj01GSeUfz2cvEvr59Kv+FkNEpSI8P+SnDMoY5l3AeUESzZWBGFOVaIh4gjLFVjJVXC10/h/6RTrZiK31jl5vmijiI4BEfgBJigDprgErRAG2CQgQfwBJ61e+1Re9Fe56MFbGzD35Ae/sERveVsQ=</latexit>

JD = 0, |M| = 0

<latexit sha1_base64="ZX/liC3suqBjwAIMXcBNbRQH3Ck=">ACBXicdZDLSsNAFIYn9VbrLepSF4NFcCEliaGtC6WgCxGECvYCTSiT6bQdOrkwMxFK2o0bX8WNC0Xc+g7ufBsnbQUV/WHg5zvnMOf8XsSokIbxoWXm5hcWl7LuZXVtfUNfXOrLsKY1LDIQt50OCMBqQmqSkWbECfI9Rhre4CytN24JFzQMbuQwIq6PegHtUoykQm197J9Dk+gcQgdOHJ8JPsYseRqPEphW8bheNy0bKL0CgYRsm0zNRYJfvIhqYiqfJgpmpbf3c6IY59EkjMkBAt04ikmyAuKWZknHNiQSKEB6hHWsoGyCfCTSZXjOG+Ih3YDbl6gYQT+n0iQb4Q9Tneme4ncthX/VWrHslt2EBlEsSYCnH3VjBmUI0hgh3KCJRsqgzCnaleI+4gjLFVwORXC16Xwf1O3Cqby13a+cjqLIwt2wB4ACYogQq4AFVQAxjcgQfwBJ61e+1Re9Fep60ZbTazDX5Ie/sEQiWgw=</latexit><latexit sha1_base64="ZX/liC3suqBjwAIMXcBNbRQH3Ck=">ACBXicdZDLSsNAFIYn9VbrLepSF4NFcCEliaGtC6WgCxGECvYCTSiT6bQdOrkwMxFK2o0bX8WNC0Xc+g7ufBsnbQUV/WHg5zvnMOf8XsSokIbxoWXm5hcWl7LuZXVtfUNfXOrLsKY1LDIQt50OCMBqQmqSkWbECfI9Rhre4CytN24JFzQMbuQwIq6PegHtUoykQm197J9Dk+gcQgdOHJ8JPsYseRqPEphW8bheNy0bKL0CgYRsm0zNRYJfvIhqYiqfJgpmpbf3c6IY59EkjMkBAt04ikmyAuKWZknHNiQSKEB6hHWsoGyCfCTSZXjOG+Ih3YDbl6gYQT+n0iQb4Q9Tneme4ncthX/VWrHslt2EBlEsSYCnH3VjBmUI0hgh3KCJRsqgzCnaleI+4gjLFVwORXC16Xwf1O3Cqby13a+cjqLIwt2wB4ACYogQq4AFVQAxjcgQfwBJ61e+1Re9Fep60ZbTazDX5Ie/sEQiWgw=</latexit><latexit sha1_base64="ZX/liC3suqBjwAIMXcBNbRQH3Ck=">ACBXicdZDLSsNAFIYn9VbrLepSF4NFcCEliaGtC6WgCxGECvYCTSiT6bQdOrkwMxFK2o0bX8WNC0Xc+g7ufBsnbQUV/WHg5zvnMOf8XsSokIbxoWXm5hcWl7LuZXVtfUNfXOrLsKY1LDIQt50OCMBqQmqSkWbECfI9Rhre4CytN24JFzQMbuQwIq6PegHtUoykQm197J9Dk+gcQgdOHJ8JPsYseRqPEphW8bheNy0bKL0CgYRsm0zNRYJfvIhqYiqfJgpmpbf3c6IY59EkjMkBAt04ikmyAuKWZknHNiQSKEB6hHWsoGyCfCTSZXjOG+Ih3YDbl6gYQT+n0iQb4Q9Tneme4ncthX/VWrHslt2EBlEsSYCnH3VjBmUI0hgh3KCJRsqgzCnaleI+4gjLFVwORXC16Xwf1O3Cqby13a+cjqLIwt2wB4ACYogQq4AFVQAxjcgQfwBJ61e+1Re9Fep60ZbTazDX5Ie/sEQiWgw=</latexit><latexit sha1_base64="ZX/liC3suqBjwAIMXcBNbRQH3Ck=">ACBXicdZDLSsNAFIYn9VbrLepSF4NFcCEliaGtC6WgCxGECvYCTSiT6bQdOrkwMxFK2o0bX8WNC0Xc+g7ufBsnbQUV/WHg5zvnMOf8XsSokIbxoWXm5hcWl7LuZXVtfUNfXOrLsKY1LDIQt50OCMBqQmqSkWbECfI9Rhre4CytN24JFzQMbuQwIq6PegHtUoykQm197J9Dk+gcQgdOHJ8JPsYseRqPEphW8bheNy0bKL0CgYRsm0zNRYJfvIhqYiqfJgpmpbf3c6IY59EkjMkBAt04ikmyAuKWZknHNiQSKEB6hHWsoGyCfCTSZXjOG+Ih3YDbl6gYQT+n0iQb4Q9Tneme4ncthX/VWrHslt2EBlEsSYCnH3VjBmUI0hgh3KCJRsqgzCnaleI+4gjLFVwORXC16Xwf1O3Cqby13a+cjqLIwt2wB4ACYogQq4AFVQAxjcgQfwBJ61e+1Re9Fep60ZbTazDX5Ie/sEQiWgw=</latexit>

JD ⌧ 1

<latexit sha1_base64="o3FylyOJ/rV9ZqAyZkIAleyMSa0=">AB8HicdVBNS8NAEJ34WetX1aOXxSJ4KkMb1IQ/iqYL9kDaUzXbTLt1swu5GKW/wosHRbz6c7z5b9y0FVT0wcDjvRlm5gUJZ0rb9oe1tLyurae28hvbm3v7Bb29psqTiWhDRLzWLYDrChngjY05y2E0lxFHDaCkYXmd+6p1KxWNzqcUL9CA8ECxnB2kh3171L1OUcOb1C0S6dVcuV0Z2ybYrjutkxK14px5yjJKhCAvUe4X3bj8maUSFJhwr1XHsRPsTLDUjnE7z3VTRBJMRHtCOoQJHVPmT2cFTdGyUPgpjaUpoNFO/T0xwpNQ4CkxnhPVQ/fYy8S+vk+qw6k+YSFJNBZkvClOdIy71GfSUo0HxuCiWTmVkSGWGKiTUZ5E8LXp+h/0nRLjuE3XrF2vogjB4dwBCfgQAVqcAV1aACBCB7gCZ4taT1aL9brvHXJWswcwA9Yb5+m7o+k</latexit><latexit sha1_base64="o3FylyOJ/rV9ZqAyZkIAleyMSa0=">AB8HicdVBNS8NAEJ34WetX1aOXxSJ4KkMb1IQ/iqYL9kDaUzXbTLt1swu5GKW/wosHRbz6c7z5b9y0FVT0wcDjvRlm5gUJZ0rb9oe1tLyurae28hvbm3v7Bb29psqTiWhDRLzWLYDrChngjY05y2E0lxFHDaCkYXmd+6p1KxWNzqcUL9CA8ECxnB2kh3171L1OUcOb1C0S6dVcuV0Z2ybYrjutkxK14px5yjJKhCAvUe4X3bj8maUSFJhwr1XHsRPsTLDUjnE7z3VTRBJMRHtCOoQJHVPmT2cFTdGyUPgpjaUpoNFO/T0xwpNQ4CkxnhPVQ/fYy8S+vk+qw6k+YSFJNBZkvClOdIy71GfSUo0HxuCiWTmVkSGWGKiTUZ5E8LXp+h/0nRLjuE3XrF2vogjB4dwBCfgQAVqcAV1aACBCB7gCZ4taT1aL9brvHXJWswcwA9Yb5+m7o+k</latexit><latexit sha1_base64="o3FylyOJ/rV9ZqAyZkIAleyMSa0=">AB8HicdVBNS8NAEJ34WetX1aOXxSJ4KkMb1IQ/iqYL9kDaUzXbTLt1swu5GKW/wosHRbz6c7z5b9y0FVT0wcDjvRlm5gUJZ0rb9oe1tLyurae28hvbm3v7Bb29psqTiWhDRLzWLYDrChngjY05y2E0lxFHDaCkYXmd+6p1KxWNzqcUL9CA8ECxnB2kh3171L1OUcOb1C0S6dVcuV0Z2ybYrjutkxK14px5yjJKhCAvUe4X3bj8maUSFJhwr1XHsRPsTLDUjnE7z3VTRBJMRHtCOoQJHVPmT2cFTdGyUPgpjaUpoNFO/T0xwpNQ4CkxnhPVQ/fYy8S+vk+qw6k+YSFJNBZkvClOdIy71GfSUo0HxuCiWTmVkSGWGKiTUZ5E8LXp+h/0nRLjuE3XrF2vogjB4dwBCfgQAVqcAV1aACBCB7gCZ4taT1aL9brvHXJWswcwA9Yb5+m7o+k</latexit><latexit sha1_base64="o3FylyOJ/rV9ZqAyZkIAleyMSa0=">AB8HicdVBNS8NAEJ34WetX1aOXxSJ4KkMb1IQ/iqYL9kDaUzXbTLt1swu5GKW/wosHRbz6c7z5b9y0FVT0wcDjvRlm5gUJZ0rb9oe1tLyurae28hvbm3v7Bb29psqTiWhDRLzWLYDrChngjY05y2E0lxFHDaCkYXmd+6p1KxWNzqcUL9CA8ECxnB2kh3171L1OUcOb1C0S6dVcuV0Z2ybYrjutkxK14px5yjJKhCAvUe4X3bj8maUSFJhwr1XHsRPsTLDUjnE7z3VTRBJMRHtCOoQJHVPmT2cFTdGyUPgpjaUpoNFO/T0xwpNQ4CkxnhPVQ/fYy8S+vk+qw6k+YSFJNBZkvClOdIy71GfSUo0HxuCiWTmVkSGWGKiTUZ5E8LXp+h/0nRLjuE3XrF2vogjB4dwBCfgQAVqcAV1aACBCB7gCZ4taT1aL9brvHXJWswcwA9Yb5+m7o+k</latexit>

SLIDE 12

Super-deep networks

In order to integrate out the auxiliary fields, the matrix has to be full rank.  But it’s a sum of Nl+1 outer products, so its rank is at most Nl+1.  The size of the matrix is the size of the dataset,   so to get rid of auxiliaries, we need to assume

Nl + 1 |M|

<latexit sha1_base64="6pebswnenEBhuregP+mBOiFvKOA=">ACEnicdVBNS0JBFL3Pvsy+rFbRZlCQJD3TNR2Ups2hUFq4BOZN46O+DmXmBPKU/0Q9o07b27aJtf6Btv6R5WlBSBy4czrmXe+9xAs6kMs13I7GwuLS8klxNra1vbG6lt3ca0g8FoXic19cO1hSzjxaV0xeh0Iil2H06YzPI395g0VkvnelRoFtO3ivsd6jGClpU5676LDUQ5ZyO730dh2sRoQzKPzybiTzpr540qpUCwhM2+aZatgxaRQLh4VkaWVGNlqxs7dA0Ctk/6wuz4JXeopwrGULcsMVDvCQjHC6SRlh5IGmAxn7Y09bBLZTuavjB1rpop4vdHkKTdWfExF2pRy5ju6Mb5TzXiz+6Tnu3GbVq7Qj5gWhoh6ZLe6FHCkfxfmgLhOUKD7SBPB9O2IDLDAROkUzqU78/R/6RyFuaX+p0TmCGJOxDBg7BgjJU4QxqUAcCt/Aj/Bk3BnPxovxOmtNGF8zu/ALxtsnxMqepA=</latexit><latexit sha1_base64="FJs0/STlhn/+jEpOe14BTkrInxE=">ACEnicdVDLSsNAFJ3UV62vqitxM7QIQqEksbR1V3TjRqlgH9CUMJlO2qGTBzMToaTFn+gvuNW9O3HrD3TrlzhpFbTogQuHc+7l3nuckFEhdX2mpVZW19Y30puZre2d3b3s/kFTBHpIEDFvC2gwRh1CcNSUj7ZAT5DmMtJzhZeK37gkXNPDv5CgkXQ/1fepSjKS7OzRjc1gARrQ6vfh2PKQHGDE4uvJ2M7m9eJ5tWyWylAv6nrFMI2EmJXSWQkaSkmQr+WswnRWG9Xt7IfVC3DkEV9ihoToGHouzHikmJGJhkrEiREeIj6pKOojzwiuvH8hQk8UoPugFX5Us4V39OxMgTYuQ5qjO5USx7ifin53hLm6Vb7cbUDyNJfLxY7EYMygAm+cAe5QRLNlIEYU7V7RAPEdYqhQzKpTvz+H/pGkWDcVvVToXYIE0OAY5cAoMUAE1cAXqoAEweACP4Ak8a1PtRXvV3hatKe1r5hD8gvb+CeaUoCo=</latexit><latexit sha1_base64="FJs0/STlhn/+jEpOe14BTkrInxE=">ACEnicdVDLSsNAFJ3UV62vqitxM7QIQqEksbR1V3TjRqlgH9CUMJlO2qGTBzMToaTFn+gvuNW9O3HrD3TrlzhpFbTogQuHc+7l3nuckFEhdX2mpVZW19Y30puZre2d3b3s/kFTBHpIEDFvC2gwRh1CcNSUj7ZAT5DmMtJzhZeK37gkXNPDv5CgkXQ/1fepSjKS7OzRjc1gARrQ6vfh2PKQHGDE4uvJ2M7m9eJ5tWyWylAv6nrFMI2EmJXSWQkaSkmQr+WswnRWG9Xt7IfVC3DkEV9ihoToGHouzHikmJGJhkrEiREeIj6pKOojzwiuvH8hQk8UoPugFX5Us4V39OxMgTYuQ5qjO5USx7ifin53hLm6Vb7cbUDyNJfLxY7EYMygAm+cAe5QRLNlIEYU7V7RAPEdYqhQzKpTvz+H/pGkWDcVvVToXYIE0OAY5cAoMUAE1cAXqoAEweACP4Ak8a1PtRXvV3hatKe1r5hD8gvb+CeaUoCo=</latexit><latexit sha1_base64="0p1fFnyZVspgUHZbSZAg9vSro=">ACEnicdVDLSsNAFJ34rPUVdSVuBosgCWJpa27ohs3SgX7gCaEyXTSDp08mJkIJS3+hL/gVvfuxK0/4NYvcdJW0KIHLhzOuZd7/FiRoU0jA9tYXFpeWU1t5Zf39jc2tZ3dpsiSjgmDRyxiLc9JAijIWlIKhlpx5ygwGOk5Q0uMr91R7igUXgrhzFxAtQLqU8xkpy9f1rl8ETaEK714MjO0CyjxFLr8YjVy8YxbNq2SqVoVE0jIpmRmxKqXTEjSVkqEAZqi7+qfdjXASkFBihoTomEYsnRxSTEj47ydCBIjPEA90lE0RAERTjp5YQyPlNKFfsRVhRJO1J8TKQqEGAae6sxuFPNeJv7pecHcZulXnZSGcSJiKeL/YRBGcEsH9ilnGDJhogzKm6HeI+4ghLlWJehfL9OfyfNK2iqfiNUaidz+LJgQNwCI6BCSqgBi5BHTQABvfgETyBZ+1Be9Fetbdp64I2m9kDv6C9fwGZu50b</latexit>

<latexit sha1_base64="ATXB1KY1F/Q7cLmvE2ApA6Rn4PI=">ACAnicbZC7SgNBFIbPxltcb1FLm8UgWIVdG23EoI1lRHOB7BJmJ7PJkLksM7NCWAIWvoKt9nZia+8z2PokTi6FJv5w4OM/53AOf5wyqo3vfzmFpeWV1bXiuruxubW9U9rda2iZKUzqWDKpWjHShFB6oYaRlqpIojHjDTjwdW437wnSlMp7swJRFHPUETipGxVjuMeR7e0h5Ho06p7Ff8ibxFCGZQvh0zx8AoNYpfYdiTNOhMEMad0O/NREOVKGYkZGbphpkiI8QD3StigQJzrKJy+PvCPrdL1EKlvCeBP390aOuNZDHtJjkxfz/fG5r+9mM9dNslZlFORZoYIPD2cZMwz0hvn4XWpItiwoQWEFbW/e7iPFMLGpubaUIL5CBahcVIJLN/45eolTFWEAziEYwjgFKpwDTWoAwYJT/AML86j8+q8Oe/T0YIz29mHP3I+fgB0npnH</latexit><latexit sha1_base64="SnAsyPDAdvkBTKvjU3iKRc6mqL8=">ACAnicbZC7SgNBFIbPeo3rLWpMxgEq7Bro40YtLGMaC6QXcLsZDYZMrOzMwKYUnK9hqYWcntvY+gtj6JE4uhSb+cODjP+dwDn+UcqaN5305C4tLyurhTV3fWNza7u4s1vXMlOE1ojkUjUjrClnCa0ZjhtpopiEXHaiPqXo37jirNZHJrBikNBe4mLGYEG2u1gkjkwQ3rCjxsF0te2RsLzYM/hdL5h3uWPn+61XbxO+hIkgmaGMKx1i3fS02Y2UY4XToBpmKSZ93KUtiwkWVIf5+OUhOrROB8VS2UoMGru/N3IstB6IyE4KbHp6tjcy/+1FYuayiU/DnCVpZmhCJofjCMj0SgP1GKEsMHFjBRzP6OSA8rTIxNzbWh+LMRzEP9uOxbvZKlQuYqAD7cABH4MJVOAKqlADAhIe4BGenHvnxXl13iajC850Zw/+yHn/AX1tmzs=</latexit><latexit sha1_base64="SnAsyPDAdvkBTKvjU3iKRc6mqL8=">ACAnicbZC7SgNBFIbPeo3rLWpMxgEq7Bro40YtLGMaC6QXcLsZDYZMrOzMwKYUnK9hqYWcntvY+gtj6JE4uhSb+cODjP+dwDn+UcqaN5305C4tLyurhTV3fWNza7u4s1vXMlOE1ojkUjUjrClnCa0ZjhtpopiEXHaiPqXo37jirNZHJrBikNBe4mLGYEG2u1gkjkwQ3rCjxsF0te2RsLzYM/hdL5h3uWPn+61XbxO+hIkgmaGMKx1i3fS02Y2UY4XToBpmKSZ93KUtiwkWVIf5+OUhOrROB8VS2UoMGru/N3IstB6IyE4KbHp6tjcy/+1FYuayiU/DnCVpZmhCJofjCMj0SgP1GKEsMHFjBRzP6OSA8rTIxNzbWh+LMRzEP9uOxbvZKlQuYqAD7cABH4MJVOAKqlADAhIe4BGenHvnxXl13iajC850Zw/+yHn/AX1tmzs=</latexit><latexit sha1_base64="3Y3vaTYyYunRp3V3DJnYrCtRH0=">ACAnicbZC7SgNBFIbPxluMt6ilzWAQrMKujZBG8uI5gLZJcxOZpMhc1lmZoWwpPMVbLW3E1tfxNYncZJsoYk/HPj4zmcwx+nBnr+19eaW19Y3OrvF3Z2d3bP6geHrWNyjShLaK40t0YG8qZpC3LKfdVFMsYk478fhm1u8Um2Ykg92ktJI4KFkCSPYOqsXxiIP79lQ4Gm/WvPr/lxoFYICalCo2a9+hwNFMkGlJRwb0wv81EY51pYRTqeVMDM0xWSMh7TnUGJBTZTPX56iM+cMUK0K2nR3P29kWNhzETEblJgOzLvZn5by8WS5dtchXlTKaZpZIsDicZR1ahWR5owDQlk8cYKZ+x2REdaYWJdaxYUSLEewCu2LeuD4zq81rot4ynACp3AOAVxCA26hCS0goOAZXuDVe/LevHfvYzFa8oqdY/gj7/MH6IeX+g=</latexit>

(this is deeply unrealistic, and not very robust. Is this assumption  even necessary? What’s wrong with auxiliary fields?) Under this assumption, we have the main result of the paper,

Q = Z [dz] exp " JD 2 X

i∈M

`(L+1(zL

i ), ti) 1

l=0

α)T (Σl)−1zl α + ln |Σl|

. (9)

SLIDE 13

Now what?

Q = Z [dz] exp " JD 2 X

i∈M

`(L+1(zL

i ), ti) 1

l=0

α)T (Σl)−1zl α + ln |Σl|

. (9)

Poole, Lahiri, Raghu, Sohl-Dickstein, and Ganguli [2016]  have a mean-field description of this ensemble from replacing

φl(zl

α)φl(zl α) ! hφl(zl α)φl(zl α)i

<latexit sha1_base64="NUTL9BNpLGTu0L4dEyKdCUjoezA=">ACVnicjVFNS8NAFHyN1tb6FfXoZVGEilASL3oUvXisYKtgYnjZbtulm03Y3Qi1+F/8L/4JvXrzqndxm/agrQcHFoaZebzHbJwJro3nvZachcXyUqW6XFtZXVvfcDe32jrNFWUtmopU3cSomeCStQw3gt1kimESC3YdD87H/vU9U5qn8soMxYm2JO8ykaK0VuGR9fifqD3ciClBkfTyYV0hgUhIlD3ByP8GVBGO3D2v4RUg8Sfkr3T3eDwCQCakfsWdFKaJ0waKlDrW9/LTDhCZTgV7LEW5JplSAfY7eWSkyYDkdFDY9k3yod0k2VfdKQv05McJE62ES2SCpq9nvbH4pxcnM5tN9yQcZnlhk6WdzNBbEtjTsmHa4YNWJoCVLF7e2E9lEhNfYnarYUf7aCedI+aviWX9p2zmCKuzALtTBh2M4hQtoQgsoPM7fMBn6aX05ZSdyiTqlKYz2/ALjvsNlxm30g=</latexit><latexit sha1_base64="y0oDneZtpdWbsMx8/UvynhCfg8=">ACVnicjVHLSgMxFE1Ha2t9jbp0E1qEilBm3Oiy6MZlBfuAznS4k6ZtMJMZkoxQh/6Ff+RP2K07t7oX08dCWxceCBzOZd7OQkTzpR2nGnO2tjMbxWK26Wd3b39A/vwqKXiVBLaJDGPZScERTkTtKmZ5rSTSApRyGk7fLiZ+e1HKhWLxb0eJ9SPYCjYgBHQRgps30tGrMerTz0eMCTEZytK9jTMfY4iCGn+H8Dch4O7IpTc+bA68Rdkq97J0/T+vjRmC/ef2YpBEVmnBQqus6ifYzkJoRTiclL1U0AfIAQ9o1VEBElZ/Na5jgU6P08SCW5gmN5+rPiQwipcZRaJIR6JFa9Wbin14YrWzWgys/YyJNRVksXiQcmxamnWM+0xSovnYECSmdsxGYEos1PlEwp7moF6R1UXMNvzPtXKMFiugElVEVuegS1dEtaqAmIugFvaMP9Jl7zX1ZeauwiFq5cwx+gXL/ga47lY</latexit><latexit sha1_base64="y0oDneZtpdWbsMx8/UvynhCfg8=">ACVnicjVHLSgMxFE1Ha2t9jbp0E1qEilBm3Oiy6MZlBfuAznS4k6ZtMJMZkoxQh/6Ff+RP2K07t7oX08dCWxceCBzOZd7OQkTzpR2nGnO2tjMbxWK26Wd3b39A/vwqKXiVBLaJDGPZScERTkTtKmZ5rSTSApRyGk7fLiZ+e1HKhWLxb0eJ9SPYCjYgBHQRgps30tGrMerTz0eMCTEZytK9jTMfY4iCGn+H8Dch4O7IpTc+bA68Rdkq97J0/T+vjRmC/ef2YpBEVmnBQqus6ifYzkJoRTiclL1U0AfIAQ9o1VEBElZ/Na5jgU6P08SCW5gmN5+rPiQwipcZRaJIR6JFa9Wbin14YrWzWgys/YyJNRVksXiQcmxamnWM+0xSovnYECSmdsxGYEos1PlEwp7moF6R1UXMNvzPtXKMFiugElVEVuegS1dEtaqAmIugFvaMP9Jl7zX1ZeauwiFq5cwx+gXL/ga47lY</latexit><latexit sha1_base64="tG2wIdvpPUQNH+2o4cYsR7YQ9nk=">ACVnicjVE9T8MwFHQDpaV8BRhZLCqkslQJC4wVLIxFoh9Sk0YvrtNaOE5kO0il6q/jT8DKxgo7wk0zQMvASZOd/f0ns5hypnSjvNasjY2y1uV6nZtZ3dv/8A+POqJOEdkjCE9kPQVHOBO1opjntp5JCHLaCx9uFn7vkUrFEnGvpyn1YxgLFjEC2kiB7XvphA1542nIAw94OoHzdQV7OsEeBzHmFP9vQObhwK47TScHXiduQeqoQDuw37xRQrKYCk04KDVwnVT7M5CaEU7nNS9TNAXyAGM6MFRATJU/y2uY4zOjHCUSPOExrn6c2IGsVLTODTJGPRErXoL8U8vjFc26+jKnzGRZpoKslwcZRyblhYd4xGTlGg+NQSIZOZ2TCYgWjzEzVTirtawTrpXjRdw+ceu6qKeKTtApaiAXaIWukVt1EPaN39IE+Sy+lL6tsVZRq1TMHKNfsOxvbAq2SQ=</latexit>

It works, but doesn’t get the fluctuations (as expected): “To overcome these issues, we pursue a more  principled solution to Eq. (9) by considering  a controlled expansion for large Nl.”

SLIDE 14

From 1 datum to 0

Except that’s not what they do. First they simplify to the case of a single input:

Q = Z [dz] exp " − 1 2 ( JD`(L+1(zL), t) + z0

αz0 α

wN −1 0 xβxβ + 2 b

l=1

αzl α

wN −1 l

l(zl−1

)l(zl−1

) + 2

l=1

Nl log(2

wN −1 l

l(zl

α)l(zl α) + 2 b)

)# . (10)

This is still too hard, so they get rid of the input and put the network 

n a circle, which necessarily requires JD = 0.

L4 L

L6 L7 L0 L1 L2 L

L({zl}) =1 2

l=0

( zl

αzl α

σ2

wN −1 l

φ(zl−1

)φ(zl−1

) + σ2

+ N log(σ2

wN −1φ(zl β)φ(zl β) + σ2 b)

) (11)

In my opinion, this is a cute interacting field theory model, but by this point has basically nothing to do with neural networks.

SLIDE 15

What’s left: sanity checks

Try a linear activation function: action is now isotropic in z, so switch to spherical coords,

L({rl}) =1 2

l=0

( (rl)2 2

wN −1(rl−1)2 + 2 b

− N log (rl)2 2

wN −1(rl)2 + 2 b

!) . (12)

Find minimum with saddle-  point approximation:

(N ≡ Nl, (rl)2 = zl

αzl α)

<latexit sha1_base64="04POswk5MbvcguJajb6LIHNrQMY=">ACKnicbZC7TgJBFIbP4g3xtmpM4GYQDRkl0YbE6KNFcFELgkLm9lhgAmzF2dmSZDwHLY+gK9gq70dsTM+iMOlQPRPJvnyn3NyzvxexJlUljUxEmvrG5tbye3Uzu7e/oF5eFSVYSwIrZCQh6LuYUk5C2hFMcVpPRIU+x6nNa9/M63XBlRIFgb3ahjRpo+7AeswgpW2XNPOlpBDH2I2QCWXnyMHZUWL51oFdIUeXQfzqIdbfAlzrpmx8tZM6C/YC8gU087ZMwCUXfPLaYck9mgCMdSNmwrUs0RFoRTscpJ5Y0wqSPu7ShMcA+lc3R7GtjdKqdNuqEQr9AoZm7PDHCvpRD39OdPlY9uVqbmv/WPH9ls+pcNkcsiGJFAzJf3Ik5UiGa5obaTFCi+FADJoLp2xHpYGJ0umdCj2agR/oVrI25rvdDrXMFcSTiANWbDhAopwC2WoAIEneIU3eDdejA9jYnzOWxPGYuYfsn4/gG/uabK</latexit><latexit sha1_base64="C/CMTN8e8mCa0ykMhZsorTO+k=">ACKnicbZDLSsNAFIYn9VbrLerSzdAitCgl6UY3QtGNq1LBXqBJw2Q6aYdOLs5MCjH0LQRfwldwq3t3xZ34IE4vi1r9YeDjP+dwzvxuxKiQhjHRMmvrG5tb2e3czu7e/oF+eNQUYcwxaeCQhbztIkEYDUhDUslIO+IE+S4jLXd4M623RoQLGgb3MomI7aN+QD2KkVSWo5vFGrTIQ0xHsOawc2jBIu+yUrcCr+CjYyEWDVCXLWHJ0QtG2ZgJ/gVzAYVq3jp7mlSTuqN/Wb0Qxz4JGZIiI5pRNJOEZcUMzLOWbEgEcJD1CcdhQHyibDT2dfG8FQ5PeiFXL1Awpm7PJEiX4jEd1Wnj+RArNam5r81/ZL1LO6VBFEsS4PliL2ZQhnCaG+xRTrBkiQKEOVW3QzxAHGp0s2pUMzVCP5Cs1I2Fd+pdK7BXFlwAvKgCExwAargFtRBA2DwDF7BG3jXrQPbaJ9zlsz2mLmGPyS9v0D4YOoUA=</latexit><latexit sha1_base64="C/CMTN8e8mCa0ykMhZsorTO+k=">ACKnicbZDLSsNAFIYn9VbrLerSzdAitCgl6UY3QtGNq1LBXqBJw2Q6aYdOLs5MCjH0LQRfwldwq3t3xZ34IE4vi1r9YeDjP+dwzvxuxKiQhjHRMmvrG5tb2e3czu7e/oF+eNQUYcwxaeCQhbztIkEYDUhDUslIO+IE+S4jLXd4M623RoQLGgb3MomI7aN+QD2KkVSWo5vFGrTIQ0xHsOawc2jBIu+yUrcCr+CjYyEWDVCXLWHJ0QtG2ZgJ/gVzAYVq3jp7mlSTuqN/Wb0Qxz4JGZIiI5pRNJOEZcUMzLOWbEgEcJD1CcdhQHyibDT2dfG8FQ5PeiFXL1Awpm7PJEiX4jEd1Wnj+RArNam5r81/ZL1LO6VBFEsS4PliL2ZQhnCaG+xRTrBkiQKEOVW3QzxAHGp0s2pUMzVCP5Cs1I2Fd+pdK7BXFlwAvKgCExwAargFtRBA2DwDF7BG3jXrQPbaJ9zlsz2mLmGPyS9v0D4YOoUA=</latexit><latexit sha1_base64="aqlVgbN0N2YKRLbVg/Aesp5akA=">ACKnicbZDLSsNAFIYn9VbrLerSzWARWpCSdKMboejGValgL9CkYTKdtkMnkzgzKdTQ5/AlfAW3undX3IkP4rTNorb+MPDxn3M4Z34/YlQqy5oamY3Nre2d7G5ub/g8Mg8PmnIMBaY1HIQtHykSMclJXVDHSigRBgc9I0x/ezerNERGShvxRjSPiBqjPaY9ipLTlmXahCh3yFNMRrHrsEjqwIDqs2CnDG/jsOYhFA9RhS1j0zLxVsuaC62CnkAepap757XRDHAeEK8yQlG3bipSbIKEoZmSc2JIoSHqE/aGjkKiHST+dcm8EI7XdgLhX5cwbm7PJGgQMpx4OvOAKmBXK3NzH9rfrCyWfWu3YTyKFaE48XiXsygCuEsN9ilgmDFxhoQFlTfDvEACYSVTjenQ7FXI1iHRrlka36w8pXbNJ4sOAPnoABscAUq4B7UQB1g8ALewDv4MF6NT2NqfC1aM0Y6cwr+yPj5BZSqpUE=</latexit>

r∗ = s Nσ2

1 − σ2

<latexit sha1_base64="0i4aulEX+xS0TqEp1GH8Ls9szNA=">ACJ3icbVDLSgMxFL3js9bXqEs3wSKIYJnpRjdCURcuRCrYB/RFJs20scnMmGSUMvQv/Al/wa3u3Ynu1C8xfSy09UDgnHPv5d4cL+JMacf5sGZm5+YXFlNL6eWV1bV1e2OzpMJYElokIQ9lxcOKchbQoma0okKRYep2Wvezqol+oVCwMrnUvonWB2wHzGcHaWE07Kxv76BjV1K3USc2XmCSXRrG2wE2vkesn7sFY3RvVb9oZJ+sMgaJOyaZ/NnNxScAFJr2d60VkljQBOlaq6TqTrCZaEU76VqsaIRJF7dp1dAC6rqyfBfbRrnBbyQ2leoNHQ/T2RYKFUT3imU2DdUZO1gflvzRMTm7V/VE9YEMWaBmS02I850iEahIZaTFKiec8QTCQztyPSwSYsbaJNm1DcyQimSmXdQ2/MumcwAgp2IYd2AMXDiEP51CAIhB4gCd4hfr0Xq13qz3UeuMNZ7Zgj+wvn4AZTOoDQ=</latexit><latexit sha1_base64="qypOkABQwGCvGkX6DiF9evQPao=">ACJ3icbVDLSgMxFM34rPU16tJNsAgiWGa60Y1Q1IULkQr2AZ12yKSZNjaZGZOMUob5C/AleAXuNW9O9GV6JeYPhbaeiBwzrn3cm+OFzEqlWV9GFPTM7Nz85mF7OLS8squbZekWEsMCnjkIWi5iFJGA1IWVHFSC0SBHGPkarXPe7XqzdESBoGl6oXkQZH7YD6FCOlLdfMi+YuPISOvBYqcXyBcHKuFW1z5HrNQprYeyN1q1Xqmjkrbw0AJ4k9IrniydXZ5+O9V3LNb6cV4piTQGpKzbVqQaCRKYkbSrBNLEiHcRW1S1zRAnMhGMvhXCre104J+KPQLFBy4vycSxKXscU93cqQ6crzWN/+teXxs/IPGgkNoliRA8X+zGDKoT90GCLCoIV62mCsKD6dog7SIeldLRZHYo9HsEkqRTytuYXOp0jMEQGbItsANsA+K4BSUQBlgcAewDN4MR6MV+PNeB+2ThmjmQ3wB8bXD8G5qb0=</latexit><latexit sha1_base64="qypOkABQwGCvGkX6DiF9evQPao=">ACJ3icbVDLSgMxFM34rPU16tJNsAgiWGa60Y1Q1IULkQr2AZ12yKSZNjaZGZOMUob5C/AleAXuNW9O9GV6JeYPhbaeiBwzrn3cm+OFzEqlWV9GFPTM7Nz85mF7OLS8squbZekWEsMCnjkIWi5iFJGA1IWVHFSC0SBHGPkarXPe7XqzdESBoGl6oXkQZH7YD6FCOlLdfMi+YuPISOvBYqcXyBcHKuFW1z5HrNQprYeyN1q1Xqmjkrbw0AJ4k9IrniydXZ5+O9V3LNb6cV4piTQGpKzbVqQaCRKYkbSrBNLEiHcRW1S1zRAnMhGMvhXCre104J+KPQLFBy4vycSxKXscU93cqQ6crzWN/+teXxs/IPGgkNoliRA8X+zGDKoT90GCLCoIV62mCsKD6dog7SIeldLRZHYo9HsEkqRTytuYXOp0jMEQGbItsANsA+K4BSUQBlgcAewDN4MR6MV+PNeB+2ThmjmQ3wB8bXD8G5qb0=</latexit><latexit sha1_base64="p4b7qlhCUk3lz09lzYDj9wbB2vY=">ACJ3icbVDLSgMxFM34rPU16tJNsAgiWGa60Y1QdONKtgHtNMhk2ba0CQzJhmlDPMX/oS/4Fb37kSX+iWm7Sy09UDgnHPv5d6cIGZUacf5tBYWl5ZXVgtrxfWNza1te2e3oaJEYlLHEYtkK0CKMCpIXVPNSCuWBPGAkWYwvBzXm/dEKhqJWz2KicdRX9CQYqSN5dtl2T2G57Cj7qRO6FEOL02ivY58oNuJUvdk1w9GJX5dskpOxPAeLmpARy1Hz7u9OLcMKJ0JghpdquE2svRVJTzEhW7CSKxAgPUZ+0DRWIE+Wlk39l8NA4PRhG0jyh4cT9PZEirtSIB6aTIz1Qs7Wx+W8t4DObdXjmpVTEiSYCTxeHCYM6guPQYI9KgjUbGYKwpOZ2iAfIhKVNtEUTijsbwTxpVMqu4TdOqXqRx1MA+AHAEXnIquAI1UAcYPIJn8AJerSfrzXq3PqatC1Y+swf+wPr6AW2KpfM=</latexit>

Expand around minimum, Fourier   transform, find EFT of fluctuations   to quadratic order:

U({✏q}) = L∆w 2

(1 + 4

w) − 22 w cos q

|✏q|2.

Stat mech works!

SLIDE 16

What’s left: sanity checks

Try a ReLU activation function: break pre-activations into + and - components Find minimum:  half of pre-activations are zero,  as may be expected Expand around minimum, find EFT  

f fluctuations to quadratic order:

U = 1 2 Z dx " (1 − 2

w + 4 w/2)(✏+(x))2 + (✏−(x))2 + 3(✏k(x))2 + 2 w✏+(x)✏−(x)

+ ✏k(x)(✏+(x) − ✏−(x)) + 2

✓@✏+(x) @x ◆2 + 2

@✏+(x) @x ✏−(x) # . (23)

This breaks symmetry ,  as expected because ReLU  distinguishes between forward/backward

✏ → −✏

<latexit sha1_base64="D1WeBl30Bcxi6ryLJUNwWFrbAs=">ACD3icbVC7SgNBFL0bX0l8rQo2NoNBsDHs2mgZtLGMYB6QDXF2MpsMmZldZmaFsOYjLG1tbcTWz/B1i9x8hA08cDAuefcy71zwoQzbTzv08ktLa+sruULxfWNza1td2e3ruNUEVojMY9VM8SaciZpzTDaTNRFIuQ0Y4uBz7jTuqNIvljRkmtC1wT7KIEWys1H3A5poxmOJAhOjk5+q45a8sjcBWiT+jJQqheTxFgCqHfcr6MYkFVQawrHWLd9LTDvDyjDC6agYpJomAxwj7YslVhQ3c4m94/QkVW6KIqVfdKgifp7IsNC6EIbafApq/nvbH4rxeKuc0mOm9nTCapoZJMF0cpR/br43BQlylKDB9agoli9nZE+lhYmyERuKPx/BIqmfln3Lr206FzBFHg7gEI7BhzOowBVUoQYE7uEJnuHFeXBenTfnfdqac2Yze/AHzsc3zAieMw=</latexit><latexit sha1_base64="TQVjEUzfzFg8bHu8h38yWtWv5H8=">ACD3icbVDLSgMxFM34bOtrVHDjJlgEN5YZN7osunFZwT6wM5RMmlDk0xIMmIZ+xEu3bpVcOlO3PoJbv0RTR+Cth4InHvOvdybE0lGtfG8D2dufmFxaTmXL6ysrq1vuJtbNZ2kCpMqTliGhHShFBqoYaRhpSEcQjRupR72zo16+J0jQRl6YvSchR9CYmSs1HJ3AiI1ZYmAgUng4U/VcoteyRsBzhJ/QorlvLy/er75qrTcz6Cd4JQTYTBDWjd9T5owQ8pQzMigEKSaSIR7qEOalgrEiQ6z0f0DuG+VNowTZ8wcKT+nsgQ17rPI9vJkenqaW8o/utFfGqziU/CjAqZGiLweHGcMmi/PgwHtqki2LC+JQgram+HuIsUwsZGWLCh+NMRzJLaUcm3/MKmcwrGyIFdsAcOgA+OQRmcgwqoAgxuwQN4BE/OnfPivDpv49Y5ZzKzDf7Aef8GxJmgUw=</latexit><latexit sha1_base64="TQVjEUzfzFg8bHu8h38yWtWv5H8=">ACD3icbVDLSgMxFM34bOtrVHDjJlgEN5YZN7osunFZwT6wM5RMmlDk0xIMmIZ+xEu3bpVcOlO3PoJbv0RTR+Cth4InHvOvdybE0lGtfG8D2dufmFxaTmXL6ysrq1vuJtbNZ2kCpMqTliGhHShFBqoYaRhpSEcQjRupR72zo16+J0jQRl6YvSchR9CYmSs1HJ3AiI1ZYmAgUng4U/VcoteyRsBzhJ/QorlvLy/er75qrTcz6Cd4JQTYTBDWjd9T5owQ8pQzMigEKSaSIR7qEOalgrEiQ6z0f0DuG+VNowTZ8wcKT+nsgQ17rPI9vJkenqaW8o/utFfGqziU/CjAqZGiLweHGcMmi/PgwHtqki2LC+JQgram+HuIsUwsZGWLCh+NMRzJLaUcm3/MKmcwrGyIFdsAcOgA+OQRmcgwqoAgxuwQN4BE/OnfPivDpv49Y5ZzKzDf7Aef8GxJmgUw=</latexit><latexit sha1_base64="b/803Nmf13LbcuIvr8dKkJu6OVM=">ACD3icbVDLSgMxFL1TX7W+RgU3boJFcGOZcaPLohuXFWwrdIaSTNtaJIZkoxQxn6Ev+BW9+7ErZ/g1i8xbUfQ1gOBc8+5l3tzopQzbTzv0yktLa+srpXKxubW9s7u5eSyeZIrRJEp6ouwhrypmkTcMp3epolhEnLaj4dXEb9TpVkib80opaHAfcliRrCxUtc9CGiqGU8kCkyCTn+qrlv1at4UaJH4BalCgUbX/Qp6CckElYZwrHXH91IT5lgZRjgdV4JM0xSTIe7TjqUSC6rDfHr/GB1bpYfiRNknDZqvydyLQeich2CmwGet6biP96kZjbOKLMGcyzQyVZLY4zjiyX5+Eg3pMUWL4yBJMFLO3IzLAChNjI6zYUPz5CBZJ6zmW37jVeuXRTxlOIQjOAEfzqEO19CAJhB4gCd4hfn0Xl13pz3WvJKWb24Q+cj29nBpyB</latexit>

r∗

+/− =

s Nσ2

2 − σ2

<latexit sha1_base64="ziwat0YyH2E79spdPYRYxAeV10k=">ACLXicbVDLSgMxFL3js9bXqEs3wSKIos4UQTdCURcuRCpYFWwdMmjU1mxiSjlGF+xJ/wF9zq3oUgIm78DdPHQqsHAuecey/35vgxZ0o7zqs1NDwyOjaem8hPTk3PzNpz82cqSiShFRLxSF74WFHOQlrRTHN6EUuKhc/pud/a79TPb6lULApPdTumNYEbIQsYwdpYnr0lr1a9dG1zPUO7qKpupE6rgcQkPTaKNQT2/KtilhbX+rOqMyzC86G0wX6S9w+KZQOro8+AKDs2Z/VekQSQUNOFbq0nViXUux1IxwmuWriaIxJi3coJeGhlhQVUu7v8vQsnHqKIikeaFGXfnRIqFUm3hm06BdVMN1jrmvzVfDGzWwU4tZWGcaBqS3uIg4UhHqBMdqjNJieZtQzCRzNyOSBObsLQJOG9CcQcj+EvOihu4ScmnT3oIQeLsAQr4MI2lOAQylABAvfwCE/wbD1YL9ab9d5rHbL6MwvwC9bXN5UNqig=</latexit><latexit sha1_base64="po+Yc4XwQAz0GkgUv5K9EJizy4=">ACLXicbVDLSgMxFM3UV62vqks3wSKI0jpTBN0IRV24EKlgH9DHkEkzbWwyMyYZpQzI27dCv6CW927ETBjb9h+lho9UDgnHPv5d4cJ2BUKtN8NRITk1PTM8nZ1Nz8wuJSenmlLP1QYFLCPvNF1UGSMOqRkqKkWogCOIOIxWne9SvV6JkNT3LlQvIA2O2h51KUZKW3Z6VzS37Gh7JxvDA1iXV0JFdVcgHJ1pRdsc2U4zH0f57EjdaBXb6YyZMweAf4k1IpnC8eXpx/2dU7Tn/Wj0NOPIUZkrJmYFqREgoihmJU/VQkgDhLmqTmqYe4kQ2osHvYrihnRZ0faGfp+DA/TkRIS5ljzu6kyPVkeO1vlvzeFjm5W734ioF4SKeHi42A0ZVD7sRwdbVBCsWE8ThAXVt0PcQTospQNO6VCs8Qj+knI+Z2l+rtM5BEMkwRpYB5vAnugAE5AEZQABrfgETyBZ+PBeDHejPdha8IYzayCXzC+vgHxk6vY</latexit><latexit sha1_base64="po+Yc4XwQAz0GkgUv5K9EJizy4=">ACLXicbVDLSgMxFM3UV62vqks3wSKI0jpTBN0IRV24EKlgH9DHkEkzbWwyMyYZpQzI27dCv6CW927ETBjb9h+lho9UDgnHPv5d4cJ2BUKtN8NRITk1PTM8nZ1Nz8wuJSenmlLP1QYFLCPvNF1UGSMOqRkqKkWogCOIOIxWne9SvV6JkNT3LlQvIA2O2h51KUZKW3Z6VzS37Gh7JxvDA1iXV0JFdVcgHJ1pRdsc2U4zH0f57EjdaBXb6YyZMweAf4k1IpnC8eXpx/2dU7Tn/Wj0NOPIUZkrJmYFqREgoihmJU/VQkgDhLmqTmqYe4kQ2osHvYrihnRZ0faGfp+DA/TkRIS5ljzu6kyPVkeO1vlvzeFjm5W734ioF4SKeHi42A0ZVD7sRwdbVBCsWE8ThAXVt0PcQTospQNO6VCs8Qj+knI+Z2l+rtM5BEMkwRpYB5vAnugAE5AEZQABrfgETyBZ+PBeDHejPdha8IYzayCXzC+vgHxk6vY</latexit><latexit sha1_base64="2QPMUzFN04r4IHhsK45Q/yD8Ec=">ACLXicbVDLSsNAFJ3UV62vqEs3g0UQpTUpgm6EohtXUsE+oGnDZDph84kcWailJAf8Sf8Bbe6dyGICzf+htM2C60eGDjn3Hu5d4XMSqVZb0Zubn5hcWl/HJhZXVtfcPc3GrIMBaY1HIQtHykCSMBqSuqGKkFQmCuMdI0xtejOvNOyIkDYMbNYpIh6N+QH2KkdKWax6L7oGbHB6VUngGHXkrVOL4AuHkSiva58j1upU0qZQyda9V6pFq2xNAP8SOyNFkKHmp9OL8QxJ4HCDEnZtq1IdRIkFMWMpAUnliRCeIj6pK1pgDiRnWTyuxTuacH/VDoFyg4cX9OJIhLOeKe7uRIDeRsbWz+W/P4zGbln3YSGkSxIgGeLvZjBlUIx9HBHhUEKzbSBGFB9e0QD5AOS+mACzoUezaCv6RKduaX1vF6nkWTx7sgF2wD2xwAqrgEtRAHWDwAJ7AM3gxHo1X4934mLbmjGxmG/yC8fUNnWSoDg=</latexit>

L({zl}) =1 2

l=0

( (zl

+)2 + (zl −)2

σ2

wN −1(zl +)2 + σ2 b

+ N log(σ2

wN −1(zl +)2 + σ2 b)

) . (18)

Can switch to spherical coords. with introduction of new field k which counts  positive/negative orthants.

SLIDE 17

What have we learned?

(my heavily subjective conclusions, not the authors’)

If you decide from the outset you want a stat mech model for a neural

network, you can get one, as long as you’re veeeeery flexible  with your definition of “neural network.” 

That said, change of variables to pre-activations seems like a useful trick.

Do we need to be afraid of auxiliary fields? Probably not (c.f. Faddeev-Popov  ghosts, very useful for computing tree-level Feynman diagrams in Yang-Mills). 

Lots of obvious follow-ups: higher-point correlators, perturbation theory in

JD, data as external sources in EFT, 1/N corrections, etc. 

More generally, is there any high-level connection between ensembles of

random networks and the behavior of a single network during training?  Can we think of training as a flow in the space of random networks? If so,  this might be a useful formalism to understand generalization.

Neural Networks as Stat Mech Systems

Motivation

What can we do with a stat mech analogy?

Feed-forward network: setting notation

{

f(x) = φL+1(W LφL(· · · φ0(W 0x + b0) · · · ) + bL)

What do we (usually) want from a neural network?

Choosing a random network

P(f) = e−L(f)/Q

Changing variables

Quick overview of the proof

{

Quick overview of the proof

Quick overview of the proof

The rest of the paper

Super-deep networks

Now what?

From 1 datum to 0

What’s left: sanity checks

What’s left: sanity checks

What have we learned?

Neural Networks  as  Stat Mech Systems

What can we do  with a stat mech analogy?

What do we (usually) want from  a neural network?