Properties of the Stochastic Approximation Schedule in the - - PowerPoint PPT Presentation

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Properties of the Stochastic Approximation Schedule in the Wang-Landau Algorithm

Pierre E. Jacob CEREMADE, Universit´ e Paris Dauphine funded by AXA research MCQMC – February 2012

joint work with Luke Bornn (UBC), Arnaud Doucet (Oxford), Pierre Del Moral (INRIA & Universit´ e de Bordeaux), Robin J. Ryder (Dauphine)

P.E.JACOB Wang-Landau 1/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Outline

1

The algorithm

2

Unsettled issues

3

Flat Histogram in finite time

4

Parallel Interacting Chains

P.E.JACOB Wang-Landau 2/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Motivation

X density

0.0 0.1 0.2 0.3 0.4 0.5 −4 −2 2 4

Figure: A normal distribution biased to get desired frequencies in specific parts of the space. Here we use φ = {75%, 25%} on {] − ∞, 0], [0, +∞[}.

P.E.JACOB Wang-Landau 3/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Motivation

Histogram of the binned coordinate

binned coordinate Density −4 −2 2 4 0.0 0.1 0.2 0.3 0.4

Figure: Normal biased to get the same frequency in each of 5 bins.

P.E.JACOB Wang-Landau 4/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Setting

Partition the state space X =

d

• i=1

Xi Desired frequencies φ = (φ1, . . . , φd) such that

• i

φi = 1 Penalized distribution ∀i ∈ {1, . . . , d} ∀x ∈ Xi πθ(x) ∝ π(x) θ(i)

P.E.JACOB Wang-Landau 5/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

First algorithm

Algorithm 1 Wang-Landau with deterministic schedule (γt)

1: Init ∀i ∈ {1, . . . , d} set θ0(i) ← 1/d. 2: Init X0 ∈ X. 3: for t = 1 to T do 4:

Sample Xt from Kθt−1(Xt−1, ·), MH kernel targeting πθt−1.

5:

Update the penalties: log θt(i) ← log θt−1(i) + γt (1 IXi(Xt) − φi)

6: end for

P.E.JACOB Wang-Landau 6/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Flat Histogram

Issues with the first version Choice of γ has a huge impact on the results. Flat Histogram Define the counters: νt(i) :=

t

• n=1

1 IXi(Xn) Flat Histogram (FH) is reached when: max

i∈{1,...,d}

• νt(i)

t − φi

• < c

P.E.JACOB Wang-Landau 7/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Flat Histogram

Idea Instead of decreasing γt at each time step t, decrease only when the Flat Histogram criterion is reached. In practice Denote by κt the number of FH criteria reached up to time t. Use γκt instead of γt at time t. If FH is reached at time t, reset νt(i) to 0 for all i.

P.E.JACOB Wang-Landau 8/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Wang–Landau with Flat Histogram

Algorithm 2 Wang-Landau with stochastic schedule (γκt)

1: Init as before: X0, θ0(i). 2: Init κ0 ← 0. 3: for t = 1 to T do 4:

Sample Xt from Kθt−1(Xt−1, ·), MH kernel targeting πθt−1.

5:

If (FH) then κt ← κt−1 + 1, otherwise κt ← κt−1.

6:

Update the penalties: log θt(i) ← log θt−1(i) + γκt (1 IXi(Xt) − φi)

7: end for

P.E.JACOB Wang-Landau 9/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Understanding the algorithm

Pros and cons . . . it works much better than the first version. . . however it is a bit tricky to analyse. Putting a label on the algorithm It is an adaptive MCMC algorithm, ie the kernel changes at every time step. Here the target distribution changes at every time step but the proposal stays the same. Between two FH, γκt stays constant, so there is no diminishing adaptation. Hence the FH version is a bit more complicated than the deterministic version.

P.E.JACOB Wang-Landau 10/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Understanding the algorithm

A reasonable first step Proof that FH is met in finite time. (under strong assumptions) Note: it means the desired frequencies are reached, when γ stays constant. ⇒ it might be a hint that the diminishing γ does not play a big part in the algorithm.

P.E.JACOB Wang-Landau 11/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

FH is met in finite time

To be sure that eventually, for any c > 0: max

i∈{1,...,d}

• νt(i)

t − φi

• < c

we want to prove: ∀i ∈ {1, . . . , d} νt(i) t

P

− − − →

t→∞ φi

P.E.JACOB Wang-Landau 12/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Various updates

Right update log θt(i) ← log θt−1(i) + γ (1 IXi(Xt) − φi) (1) Wrong update θt(i) ← θt−1(i) [1 + γ (1 IXi(Xt) − φi)] ⇔ log θt(i) ← log θt−1(i) + log [1 + γ (1 IXi(Xt) − φi)] (2) (actually not wrong if ∀i φi = 1

d )

P.E.JACOB Wang-Landau 13/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Assumptions

From now on, there are only two bins: d = 2. Additionally: Assumption The bins are not empty with respect to µ and π: ∀i ∈ {1, 2} µ(Xi) > 0 and π(Xi) > 0 Assumption The state space X is compact.

P.E.JACOB Wang-Landau 14/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Assumptions

Assumption The proposition distribution Q(x, y) is such that: ∃qmin > 0 ∀x ∈ X ∀y ∈ X Q(x, y) > qmin Assumption The MH acceptance ratio is bounded from both sides: ∃m > 0 ∃M > 0 ∀x ∈ X ∀y ∈ X m < π(y) π(x) Q(y, x) Q(x, y) < M

P.E.JACOB Wang-Landau 15/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Theorem

Theorem Consider the sequence of penalties θt introduced in the WL

• algorithm. We define:

Zt = log θt(1) θt(2) = log θt(1) − log θt(2) Then: Zt t

L1

− − − →

t→∞ 0

and consequently, with update (1) (FH) is reached in finite time for any precision threshold c, whereas this is not guaranteed for update (2).

P.E.JACOB Wang-Landau 16/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Consequence

Recall νt(i) :=

t

• n=1

1 IXi(Xn) Using update (1), and starting from Z0 = 0: Zt = log θt(1) − log θt(2) = (νt(1)γ (1 − φ1) − (t − νt(1))γφ1) − (νt(2)γ (1 − φ2) − (t − νt(2))γφ2) = νt(1) (2γ) − t (2γφ1) using νt(1) + νt(2) = t and φ1 + φ2 = 1. Hence if Zt

t L1

− − − →

t→∞ 0 then

νt(1) t

L1

− − − →

t→∞ φ1

P.E.JACOB Wang-Landau 17/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Proof

time

• Zs

Zs+T Z ~

s+T ~ 5 10 15 20 25 30 35

Figure: We prove that Zt returns below a given horizontal bar whenever it goes above it, and it does so in finite time. It then implies Zt/t → 0.

P.E.JACOB Wang-Landau 18/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Parallel Interacting Chains

A parallel version of the algorithm runs N chains in parallel (see e.g. F.Liang, JSP 2006). Target the same distribution Each new value (X (k)

t

) is drawn from a MH kernel Kθt−1(X (k)

t−1, ·)

using the same penalties (θt). Interaction between chains To update θt use an average: 1 N

N

• k=1

1 IXi(X (k)

t

) instead of 1 IXi(Xt).

P.E.JACOB Wang-Landau 19/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Parallel Interacting Chains

Reaching Flat Histogram

iterations #FH

10 20 30 40 2000 4000 6000 8000 10000

N = 1 N = 10 N = 100

P.E.JACOB Wang-Landau 20/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Parallel Interacting Chains

Stabilization of the log penalties

iterations value

−10 −5 5 10 2000 4000 6000 8000 10000

Figure: log θt against t, for N = 1

P.E.JACOB Wang-Landau 21/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Parallel Interacting Chains

Stabilization of the log penalties

iterations value

−10 −5 5 10 2000 4000 6000 8000 10000

Figure: log θt against t, for N = 10

P.E.JACOB Wang-Landau 22/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Parallel Interacting Chains

Stabilization of the log penalties

iterations value

−10 −5 5 10 2000 4000 6000 8000 10000

Figure: log θt against t, for N = 100

P.E.JACOB Wang-Landau 23/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Removing the schedule

Desired frequencies Under some assumptions, we can prove that for fixed γ we obtain the desired frequencies. Behaviour of the penalties For fixed γ, θt does not converge but seems stable, and its variations decrease with the number of chains. Time averages Does 1

t

t

s=1 log θs(i) converge to something?

Does 1

t

t

s=1 θs(i) converge to something?

To something useful?

P.E.JACOB Wang-Landau 24/ 25

The algorithm Unsettled issues Flat Histogram in finite time Parallel Interacting Chains

Bibliography

Atchad´ e, Y. and Liu, J. (2010). The Wang-Landau algorithm in general state spaces: applications and convergence analysis. Statistica Sinica, 20:209–233. Wang, F. and Landau, D. (2001). Determining the density of states for classical statistical models: A random walk algorithm to produce a flat histogram. Physical Review E, 64(5):56101. An Adaptive Interacting Wang-Landau Algorithm for Automatic Density Exploration, with L. Bornn, P. Del Moral, A.

Doucet.

The Wang-Landau algorithm reaches the Flat Histogram criterion in finite time, with R. Ryder.

P.E.JACOB Wang-Landau 25/ 25