Applications of large deviation theory Hugo Touchette National Institute for Theoretical Physics (NITheP) Stellenbosch, South Africa Maties Machine Learning 20 April 2018 Hugo Touchette (NITheP) Large deviations April 2018 1 / 10 Overview Goal Statistics Explain how deterministic behavior arises from randomness Probability • Stochastic processes , Large Simulations • Many interacting components deviations • Rare events Statistical • Typical or generic events Physics • Emergent determinism Stochastic processes Hugo Touchette (NITheP) Large deviations April 2018 2 / 10
Large deviation theory in one slide S n S n ( � X ) Macro X 2 X 1 � X = ( X 1 , X 2 , . . . , X n ) Micro Large deviation principle (LDP) p ( S = s ) n P ( S n = s ) ≈ e − nI ( s ) n = 500 • I ( s ) = rate function I ( s ) • Exponentially rare events n = 100 • Typical value: I ( s ∗ ) = 0 n = 10 • Concentration of probability µ s Hugo Touchette (NITheP) Large deviations April 2018 3 / 10 Coin tossing � — n = 5 H T T H T · · · — n = 25 � — n = 50 1 0 0 1 0 · · · � � ( � ) — n = 100 � · · · X 1 X 2 X 3 X 4 X 5 � � n ��� ��� ��� ��� ��� ��� S n = # heads = 1 � � X i n n ��� i =1 ��� ��� ��� � ( � ) ��� LDP ��� ��� ��� P ( S n = s ) ≈ e − nI ( s ) ��� ��� ��� ��� ��� ��� � Λ n I ( s ) = s log s + (1 − s ) log(1 − s ) + log 2 Typical sequences Typical # { � X : S n = 0 . 5 } ≈ 2 n Atypical Hugo Touchette (NITheP) Large deviations April 2018 4 / 10
Gaussian vectors and hyperspheres • Gaussian vector in n -dim: � X = ( X 1 , X 2 , . . . , X n ) • X i ∼ N (0 , 1) iid ∼ n 1 / 2 • Rescaled “radius”: ∼ n 1 / 4 n S n = 1 � X 2 i n i =1 LDP Typicality • Most points lie on surface P ( S n = s ) ≈ e − nI ( s ) • Asymptotically uniform • Typical value: S n → 1 • Volume concentrated near • Exponential concentration surface as n → ∞ Hugo Touchette (NITheP) Large deviations April 2018 5 / 10 Statistical physics • Total energy: N v 2 � N ∼ 10 23 i U N = 2 m , i =1 • Velocity distribution: L N ( v ) = # particles v i ∈ [ v , v + ∆ v ] N ∆ v LDP P ( L N = ρ ) ≈ e − NI ( ρ ) ρ ( v ) • Equilibrium distribution: − mv 2 ρ ∗ ( v ) = c v 2 e 2 kB T v • Maxwell’s distribution Hugo Touchette (NITheP) Large deviations April 2018 6 / 10
Spin glasses • Energy: � � H = − J ij σ i σ j + h σ i ���� ij i disorder • Find min energy (ground state) • Count number Ω N ( u ) local min with given energy LDPs Typicality • Exponentially many Ω N ( u ) ≈ e N Σ( u ) metastable states • Most critical points are • Σ( u ) = entropy saddles in high dim Hugo Touchette (NITheP) Large deviations April 2018 7 / 10 Problems from machine learning • Neurons: { σ i } • Weights: { w ij } • Cost function: C (input , output , { w ij } ) Practical Fundamental • Number of metastable states • Why ML works at all? • Typicality of data • Generic attractors • Overfitting fluctuations • Represent typical features Hugo Touchette (NITheP) Large deviations April 2018 8 / 10
Other applications Λ n • Information theory • Random graphs Typical • Markov process: X t , S T [ x ] • Signal analysis • Statistical physics Atypical • Phase transitions • Full space is huge • Quantum systems • Most states “look” the same • ... F. den Hollander, Large Deviations , AMS, 2000 H. Touchette, The large deviation approach to statistical mechanics, Physics Reports 478 , 2009 www.physics.sun.ac.za/~htouchette Hugo Touchette (NITheP) Large deviations April 2018 9 / 10 Current research: LDs of Markov processes • Process: { X t } T t =0 x H t L • Observable: A T [ x ] LDP t P ( A T = a ) ≈ e − TI ( a ) P H A T = a L Problems • Predict how rare fluctuations happen a • Effective process describing fluctuations x ( t ) • Related to non-Hermitian spectral problem t Hugo Touchette (NITheP) Large deviations April 2018 10 / 10
Recommend
More recommend
