Kullback-Leibler Designs Astrid JOURDAN Jessica FRANCO - - PowerPoint PPT Presentation
Kullback-Leibler Designs Astrid JOURDAN Jessica FRANCO ENBIS 2009 / Saint-Etienne Contents Contents Introduction Kullback-Leibler divergence Estimation by a Monte-Carlo method Estimation by a Monte-Carlo method
ENBIS 2009 / Saint-Etienne
Introduction Kullback-Leibler divergence
Estimation by a Monte-Carlo method
Contents
ENBIS 2009 / Saint-Etienne
2
Estimation by a Monte-Carlo method Design comparison Conclusion
Introduction
Input parameters Outputs
Physical experimentation is impossible Mathematical Models
ENBIS 2009 / Saint-Etienne
3
Computer code
Outputs
x y(x)
Time-consuming
Metamodel Sensitivity Analysis Optimization Uncertainty Quantification
simulations
Introduction
the design on to a subset of parameters (non- collapsing)
experimental region Space filling designs
ENBIS 2009 / Saint-Etienne
4
experimental region
models Exploratory designs Goal : fill up the space in uniform fashion with the design points
ENBIS 2009 / Saint-Etienne
Introduction Kullback-Leibler divergence Estimation by a Monte Carlo method Design comparison Conclusion
Kullback Leibler Divergence
Suppose that the design points X1,...,Xn, are n independent
absolutely continuous density function f select the design points in such a way as to
ENBIS 2009 / Saint-Etienne
6
∫
= dx ) x ( ) x ( ln ) x ( ) , ( D g f f g f The Kullback-Leibler (KL) divergence measures the difference between two density functions f and g (with f << g) select the design points in such a way as to have the density function “close” to the uniform density function.
Kullback Leibler Divergence
(it is not symmetric, it does not satisfy the triangle inequality)
D( f , g ) = 0 ⇒ f = g p.p. If {P ,…,P } is a sequence of distributions then
ENBIS 2009 / Saint-Etienne
7
If {P1,…,Pn} is a sequence of distributions then Minimizing the KL divergence Design space = unit cube P P P P
n n n n variation Total divergence KL ∞ + → ∞ + →
→ ⇒ →
Kullback Leibler Divergence
( ) [ ]
f f f f H dx ) x ( ln ) x ( ) ( D − = = ∫ If g is the uniform density function then where H( f ) is the Shannon entropy Minimizing the KL divergence ⇔ Maximizing the entropy
ENBIS 2009 / Saint-Etienne
8
If f is supported by [0,1]d, one always has H(f)≤0 and the maximum value of H(f), zero, being uniquely attained by the uniform density.
Using an exchange algorithm to build an “optimal” design Entropy estimation
ENBIS 2009 / Saint-Etienne
Introduction Kullback Leibler divergence Estimation by a Monte Carlo method Design comparison Conclusion
Estimation by a Monte Carlo method
[ ] ( ) ( ) [ ]
) x ( ln E dx ) x ( ln ) x ( H
P
f f f f
f
− = − = ∫ The entropy can be written as an expectation The Monte Carlo method (MC) provides a unbiased and consistent estimate of the entropy
ENBIS 2009 / Saint-Etienne
10
Estimation by a Monte Carlo method
the unknown density function f is replaced by its kernel density estimate (Ahmad and Lin, 1976) estimate of the entropy
∑
=
− =
n 1 i i)
X ( ln n 1 ) X ( H ˆ f where X1,…,Xn are the design points.
Estimation by a Monte Carlo method
Joe (1989) obtained asymptotic bias and variance terms for the estimator
∑
=
− =
n 1 i i)
X ( ˆ ln n 1 ) X ( H ˆ f where is the kernel estimate, f ˆ
ENBIS 2009 / Saint-Etienne
11
Estimation by a Monte Carlo method
∑
=
− =
n 1 i i d
h X x nh 1 ) x ( ˆ K f ∀x∈[0,1]d, The bias depends on the size n, the dimension d and the bandwidth h fix the bias during the exchange algorithm
Estimation by a Monte Carlo method
The bandwidth h plays an important role in the estimation
h=0.1 h=0.4
ENBIS 2009 / Saint-Etienne
12
Estimation by a Monte Carlo method
) 4 d /( 1
n 1 12 1 h ˆ
+
=
Standard deviation
distribution
⇒ Scott’s rule
) 4 d /( 1 j j
n 1 ˆ h ˆ
+
σ =
j=1,…,d
Estimation by a Monte Carlo method
the choice of the kernel function is much less important − π =
− 2 2 d 2 / d
z s 2 1 exp s ) 2 ( ) z ( K Multidimensional Gaussian function
ENBIS 2009 / Saint-Etienne
13
Estimation by a Monte Carlo method
Remark : is no more supported by [0,1]d f ˆ where h X X z
j i −
= i,j=1,…,n
(d=10 and n=100 : )
Epanechnikov, uniform,… kernel functions are not desirable
2 2
h / d , ∈ z
[ ]
7 . 231 ;
2 ∈
z
Estimation by a Monte Carlo method
0.8
0.0 Entropie
Entropie
Entropy Entropy ENBIS 2009 / Saint-Etienne
14
Estimation by a Monte Carlo method
500 1000 1500 2000 2500 3000
Taille de l'échantillon 100 200 300 400 500
Nombre d'échanges
The entropy estimation converges slowly towards 0
Design size n E E Number of exchanges
The exchange algorithm converges rapidly
d=3 n=30 d=3
ENBIS 2009 / Saint-Etienne
Introduction Kullback-Leibler divergence Estimation by a Monte Carlo method Design comparison Conclusion
Design comparison
0.2 0.4 0.6 0.8 1.0
Plan initial
0.2 0.4 0.6 0.8 1.0
Plan final
d=2 n=20
ENBIS 2009 / Saint-Etienne
16
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Plan initial
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Plan final
Initial design KL design
0.8 1.0 0.8 1.0
the design points will generally lie on the boundary of the design space, especially for small size n
Design comparison
ENBIS 2009 / Saint-Etienne
17
2 4 6 8 10 0.0 0.2 0.4 0.6 Axes Projections 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 X1 X2
Projections on each dimension Projections on 2D plane X1X2 d=10 n=100
between the design points (Johnson et al., 1990), ) x , x ( d min
j i n j i 1 ≤ < ≤
a covariance matrix (Shewry & Wynn, 1987),
ENBIS 2009 / Saint-Etienne
18
− θ − =
= d 1 k p k j k i k j i
x x exp ) x , x ( R
between charged particles Audze-Eglais (1977) criterion (AE) minimizes
− = + = 1 n 1 i n 1 i j 2 j i
) x , x ( d 1 Strauss designs (Strauss) built with a MCMC method (Franco, 2008)
Distance criteria
quantify how the points fill up the space
0.10 0.12
Cov
Dmax
0.9 1.0
Maximin
Maximin KL Dmax Strauss
Design comparison
ENBIS 2009 / Saint-Etienne
19
Maximin KL Dmax Strauss AE 0.02 0.04 0.06 0.08
Maximin KL Dmax Strauss AE
The cover measure calculates the difference between the design and a uniform mesh (min)
Maximin KL Dmax Strauss AE 0.6 0.7 0.8
AE
The Maximin criterion maximizes the minimal distance between the design points (max)
Uniformity criteria
Measure how close points being uniformly distributed
4.8e-06 5.0e-06 5.2e-06 5.4e-06
DL2
Design comparison
KL
Maximin KL Dmax Strauss AE
KL
ENBIS 2009 / Saint-Etienne
20
Maximin KL Dmax Strauss AE 4.0e-06 4.2e-06 4.4e-06 4.6e-06 4.8e-06
Maximin KL Dmax Strauss AE
The discrepancy measures the difference between the empirical cumulative distribution of the design points and the uniform one (min)
Maximin KL Dmax Strauss AE
zoom
Maximin KL Dmax
KL divergence (max)
ENBIS 2009 / Saint-Etienne
Introduction Kullback-Leibler divergence Estimation by a Monte Carlo method Design comparison Conclusion
Conclusion
Results
ENBIS 2009 / Saint-Etienne
22
Outlooks
+ support of f)
by MST)
estimation : an overview. Int. J. Math. Stat. Sci., 6(1) 17-39.
codes de calculs simulant des phénomènes complexes. Thèse présentée à l’Ecole Nationale Supérieure des Mines de Saint-Etienne
https://people.scs.fsu.edu/~burkardt/pdf/ptmeas.pdf
ENBIS 2009 / Saint-Etienne
23
308
John Wiley & Sons, New York, Chichester
Hall, London
170