Statistical inference for R enyi entropy of integer order David K - - PowerPoint PPT Presentation

Statistical inference for R´ enyi entropy of integer

• rder

David K¨ allberg August 23, 2010

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Outline

Introduction Measures of uncertainty Estimation of entropy Numerical experiment More results Conclusion

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Coauthor: Oleg Seleznjev

Department of Mathematics and Mathematical Statistics Ume˚ a university

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Introduction

A system described by probability distribution P. Only partial information about P is available, e.g. the covariance matrix. A measure of uncertainty (entropy) in P. What P should we use if any?

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Why entropy?

The entropy maximization principle: choose P, satisfying given constraints, with maximum uncertainty. Objectivity: we don’t use more information than we have.

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Measures of uncertainty. The Shannon entropy.

Discrete P = {p(k), k ∈ D} h1(P) := −

• k

p(k) log p(k) Continuous P with density p(x), x ∈ Rd h1(P) := −

• Rd log (p(x))p(x)dx

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Measures of uncertainty.The R´ enyi entropy

A class of entropies. Of order s ≥ 0 given by Discrete P = {p(k), k ∈ D} hs(P) := 1 1 − s log (

• k

p(k)s), s = 1 Continuos P with density p(x), x ∈ Rd hs(P) := 1 1 − s log (

• Rd p(x)sdx),

s = 1

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Motivation

The R´ enyi entropy satisfies axioms on how a measure of uncertainty should behave, R´ enyi(1961,1970). For both discrete and continuous P, the R´ enyi entropy is a generalization of the Shannon entropy, lim

q→1 hq(P) = h1(P)

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Problem

Non-parametric estimation of integer order R´ enyientropy, for discrete and continuous multivariate P, from sample {X1, . . . Xn}

• f P-i.i.d. observations.

Estimators of entropy are widely used. Distribution identification problems (Student-r distributions). Average case analysis for random databases. Clustering.

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Overview of R´ enyi entropy estimation

Consistency of nearest neighbor estimators for any s, Leonenko et al. (2008). Only for continuous entropy. Consistency and asymptotic normality for quadratic case s=2, Leonenko and Seleznjev (2010).

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Notation

I(C) indicator function of event C. Ss,n set of all s-subsets of {1, . . . , n}.

• (s,n) is summation over Ss,n.

d(x, y) the Euclidean distance in Rd Bǫ(x) := {y : d(x, y) ≤ ǫ} ball of radius ǫ with center x. bǫ volume of Bǫ(x) pǫ(x) := P(X ∈ Bǫ(x)) the ǫ-ball probability at x

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Estimation

Method relies on estimating functional qs := E(ps−1(X)) =   

• k p(k)s

(Discrete)

• Rd p(x)sdx

(Continuous)

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Estimation, discrete case

An estimator of qs, Qs,n := s n −1

(s,n)

ψ(S), where ψ(S) := 1 s

• i∈S

I(Xi = Xj, ∀j ∈ S). Hs,n :=

1 1−s log(max( ˜

Qs,n, 1/n)) estimator of hs. Qs,n is a U-statistic, so properties follows from conventional theory.

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Estimation, continuous case

A U-statistic estimator of qǫ,s := E(pǫ(X)s−1), Qs,n := s n −1

(s,n)

ψ(S), where ψ(S) := 1 s

• i∈S

I(d(Xi, Xj) ≤ ǫ, ∀j ∈ S). ˜ Qs,n := Qs,n/bǫ(d)s−1 asymptotically unbiased estimator of qs if ǫ = ǫ(n) → 0 as n → ∞. Hs,n :=

1 1−s log(max( ˜

Qs,n, 1/n)) estimator of hs.

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Smoothness conditions

Denote by H(α)(K), 0 < α ≤ 2, K > 0, a linear space of continuous in Rd functions satisfying α-H¨

• lder condition if

0 < α ≤ 1 or if 1 < α ≤ 2 with continuous partial derivatives satisfying (α − 1)-H¨

• lder condition with constant K.

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Asymptotics, continuous case

ǫ = ǫ(n) → 0 as n → ∞. L(n) > 0, n ≥ 1, a slowly varying function as n → ∞. Ks,n := max(s2( ˜ Q2s−1,n − ˜ Q2

s,n), 1/n) consistent estimator of

the asymptotic variance. Theorem Suppose that p(x) is bounded and continuous. Let nǫd → a for some 0 < a ≤ ∞. (i) Then Hs,n is a consistent estimator of hs. (ii) Let p(x)s−1 ∈ H(α)(K) for some d/2 < α ≤ 2. If ǫ ∼ L(n)n−1/d and a = ∞, then √n ˜ Qs,n(1 − s)

• Ks,n

(Hs,n − hs) D → N(0, 1) as n → ∞.

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Numerical experiment

χ2 distribution, 4 degrees of freedom. h3 = − 1

2 log (q3),

where q3 = 1/54. 500 simulations, each of size n = 1000, ǫ = 1/4. Quantile plot and histogram supports standard normality.

• Remark. Choice of ǫ in a practical situation remains an open

problem.

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Figures

Histogram of x

x Standard normal density −3 −2 −1 1 2 0.0 0.1 0.2 0.3 0.4

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Figures

−3 −2 −1 1 2 3 −2 −1 1 2

Normal Q−Q Plot

Theoretical Quantiles Sample Quantiles

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

More results

Estimation of (quadratic) R´ enyi entropy from m-dependent sample. Two different distributions PX and PY . Inference for

...functionals of type

• Rd pX(x)s1pY (x)s2dx,

s1, s2 ∈ N+. ...statistical distances (Bregman)

Discrete: D2 :=

• k

(pX(k) − pY (k))2 Continuous: D2 :=

• Rd (pX(x) − pY (x))2dx

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

Conclusion

Asymptotically normal estimates possible for R´ enyi entropy of integer order.

David K¨ allberg Statistical inference for R´ enyi entropy of integer order

References

Leonenko, N. ,Pronzato, L. ,Savani, V. (1982). A class of R´ enyi information estimators for multidimensional densities, Annals of Statistics 36 2153-2182. Leonenko, N. and Seleznjev, O. (2010). Statistical inference for ǫ-entropy and quadratic R´ enyientropy, J. Multivariate Analysis 101, Issue 9, 1981-1994. R´ enyi, A.(1961). On measures of information and entropy Proceedings of the 4th Berkeley Symposium on Mathematics, Statistics and Probability 1960, 547-561. Renyi, A. Probability theory, North-Holland Publishing Company 1970

David K¨ allberg Statistical inference for R´ enyi entropy of integer order