Applications of large deviation theory Hugo Touchette National - - PDF document

Applications of large deviation theory

Hugo Touchette

National Institute for Theoretical Physics (NITheP) Stellenbosch, South Africa

Maties Machine Learning 20 April 2018

Hugo Touchette (NITheP) Large deviations April 2018 1 / 10

Overview

,

Large deviations

Statistics Probability Statistical Physics Stochastic processes Simulations

Goal

Explain how deterministic behavior arises from randomness

• Stochastic processes
• Many interacting components
• Rare events
• Typical or generic events
• Emergent determinism

Hugo Touchette (NITheP) Large deviations April 2018 2 / 10

Large deviation theory in one slide

Sn( X) Macro

• X = (X1, X2, . . . , Xn)

Micro

X1 X2 Sn

Large deviation principle (LDP)

P(Sn = s) ≈ e−nI(s)

• I(s) = rate function
• Exponentially rare events
• Typical value: I(s∗) = 0
• Concentration of probability

I(s) s n=10 n=100 n=500 µ p(S =s)

n

Hugo Touchette (NITheP) Large deviations April 2018 3 / 10

Coin tossing

H T T H T · · · 1 1 · · · X1 X2 X3 X4 X5 · · · Sn = # heads n = 1 n

n

• i=1

Xi

LDP

P(Sn = s) ≈ e−nI(s) I(s) = s log s + (1 − s) log(1 − s) + log 2

Typical sequences

#{ X : Sn = 0.5} ≈ 2n

• ()

— n=5 — n=25 — n=50 — n=100

• ()

Λn Typical Atypical Hugo Touchette (NITheP) Large deviations April 2018 4 / 10

Gaussian vectors and hyperspheres

• Gaussian vector in n-dim:
• X = (X1, X2, . . . , Xn)
• Xi ∼ N(0, 1) iid
• Rescaled “radius”:

Sn = 1 n

n

• i=1

X 2

i

∼ n1/2 ∼ n1/4

LDP

P(Sn = s) ≈ e−nI(s)

• Typical value: Sn → 1
• Exponential concentration

Typicality

• Most points lie on surface
• Asymptotically uniform
• Volume concentrated near

surface as n → ∞

Hugo Touchette (NITheP) Large deviations April 2018 5 / 10

Statistical physics

• Total energy:

UN =

N

• i=1

v2

i

2m, N ∼ 1023

• Velocity distribution:

LN(v) = # particles vi ∈ [v, v + ∆v] N∆v

LDP

P(LN = ρ) ≈ e−NI(ρ)

• Equilibrium distribution:

ρ∗(v) = c v2e

− mv2

2kB T

• Maxwell’s distribution

v ρ(v) Hugo Touchette (NITheP) Large deviations April 2018 6 / 10

Spin glasses

• Energy:

H = −

• ij

Jij

• disorder

σiσj + h

• i

σi

• Find min energy (ground state)
• Count number ΩN(u) local min

with given energy

LDPs

ΩN(u) ≈ eNΣ(u)

• Σ(u) = entropy

Typicality

• Exponentially many

metastable states

• Most critical points are

saddles in high dim

Hugo Touchette (NITheP) Large deviations April 2018 7 / 10

Problems from machine learning

• Neurons: {σi}
• Weights: {wij}
• Cost function:

C(input, output, {wij})

Practical

• Number of metastable states
• Typicality of data
• Overfitting fluctuations

Fundamental

• Why ML works at all?
• Generic attractors
• Represent typical features

Hugo Touchette (NITheP) Large deviations April 2018 8 / 10

Other applications

• Information theory
• Random graphs
• Markov process: Xt, ST[x]
• Signal analysis
• Statistical physics
• Phase transitions
• Quantum systems
• ...

Λn Typical Atypical

• Full space is huge
• Most states “look” the same
• F. den Hollander, Large Deviations, AMS, 2000
• H. Touchette, The large deviation approach to statistical mechanics,

Physics Reports 478, 2009 www.physics.sun.ac.za/~htouchette

Hugo Touchette (NITheP) Large deviations April 2018 9 / 10

Current research: LDs of Markov processes

• Process: {Xt}T

t=0

• Observable: AT[x]

LDP

P(AT = a) ≈ e−TI(a)

Problems

• Predict how rare fluctuations

happen

• Effective process describing

fluctuations

• Related to non-Hermitian

spectral problem

t xHtL a PHAT = aL t x(t)

Hugo Touchette (NITheP) Large deviations April 2018 10 / 10