Asymptotics for Empirical Process and Bootstrap Marquis Hou - - PowerPoint PPT Presentation

Asymptotics for Empirical Process and Bootstrap

Marquis Hou

University of California j7hou@ucsd.edu

Marquis Hou (UCSD) Learning Proofs 1 / 16

Overview

1

Introduction

2

Empirical Process on R Glivenko-Cantelli Theorem C` adl` ag space and Donsker Theorem Weak Convergence in l∞(R)

3

Empirical Process in General Sample Space P-Glivenko-Cantelli and P-Donsker Measurability and P-Donsker Class

4

Empirical Bootstrap Weak Convergence with Donsker Class Functional δ-Method

Marquis Hou (UCSD) Learning Proofs 2 / 16

Reference

Aad van der Vaart, Asymptotic Statistics, Ch. 19 and Ch. 23. Cambridge University Press, 1998 Aad van der Vaart, Jon Wellner, Weak Convergence and Empirical

• Processes. Springer, 1996

Evarist Gin´ e, Joel Zinn, Some Limit Theorems for Empirical

• Processes. The Annals of Probablity Vol. 12, No. 4, 1984

Evarist Gin´ e, Joel Zinn, Necessary Conditions for the Bootstrap of the

• Mean. The Annals of Statistics Vol. 17, No. 2, 1989

Evarist Gin´ e, Joel Zinn, Bootstrapping General Empirical Measures. The Annals of Probability Vol. 18, No. 2, 1990

Marquis Hou (UCSD) Learning Proofs 3 / 16

Introduction

Empirical Measure and Bootstrap Measure

Empirical cumulative distribution function: Fn(x) = 1 n

n

• i=1

χ[Xi,+∞)(x) = 1 n

n

• i=1

I(Xi ≤ x) Empirical measure: Pn(ω) = 1 n

n

• i=1

δXi(ω), ω ∈ (Ω∞, P∞, P∞) Bootstrap measure: P∗

n(ω, σ) = 1

n

n

• i=1

δX ∗

i (ω,σ) = 1

n

n

• i=1

δXσ(ω) σ ∼ Multinomial(n) with uniform pi

Marquis Hou (UCSD) Learning Proofs 4 / 16

Empirical Process on R Glivenko-Cantelli Theorem

Glivenko-Cantelli Theorem on R

Theorem (Glivenko-Cantelli) Fn − F∞

a.s.

− − → 0. Proof by partition, pick bigger jumps of F(x) as cut points.

Marquis Hou (UCSD) Learning Proofs 5 / 16

Empirical Process on R C` adl` ag space and Donsker Theorem

C` adl` ag space and Donsker Theorem

C` adl` ag space D[−∞, +∞], right continuous functions with left limits. Skorokhod metric: σ(f , g) = inf

λ∈Λ max λ − I, f − g ◦ λ

Λ is the set of all strictly increasing continuous bijection of [−∞, +∞]. Theorem (Donsker) In Skorokhod topology of C` adl` ag space D[−∞, +∞], √n(Fn − F) L − → B ◦ F where B is a Brownian bridge.

Marquis Hou (UCSD) Learning Proofs 6 / 16

Empirical Process on R Weak Convergence in l∞(R)

Weak Convergence in l∞(R)

Fact: Fn and Gn = √n(Fn − F) are not Borel measurable (Pn → B(l∞(R))). l∞(R) is neither compact nor separable. Thus, Dudley and Hoffman-Jørgensen developed the extended theory of weak convergence. Definition (Outer expectation) E∗T(P) = inf{EU : U ≥ T, Uextended r.v and EU =

• UdP exists}

Definition (Weak Convergence) Gn → G in l∞[0, 1]. For all bounded continuous h : l∞[0, 1] → R, E∗h(Gn) =→ Eh(G)

Marquis Hou (UCSD) Learning Proofs 7 / 16

Empirical Process on R Weak Convergence in l∞(R)

Second Donsker Theorem

Theorem (Donsker) If F is continuous, then Gn converges weakly in l∞(R) to B ◦ F, a tight process concentrating on a complete separable subspace of l∞(R).

Marquis Hou (UCSD) Learning Proofs 8 / 16

Empirical Process in General Sample Space P-Glivenko-Cantelli and P-Donsker

Empirical Process in General Sample Space

No more c.d.f. Fn(.) and F(.), all in terms of measure Pn and P For a measurable function f : Ω → R, Pnf = 1 n

• i=1

nf (Xi), Pf =

• fdP

No proper extension to C` adl` ag and Skorokhod, but l∞(F), where F is a class of functions.

Marquis Hou (UCSD) Learning Proofs 9 / 16

Empirical Process in General Sample Space P-Glivenko-Cantelli and P-Donsker

P-Glivenko-Cantelli and P-Donsker

Suppose F is a class of measurable functions. Definition (P-Glivenko-Cantelli) Pnf − Pf F = sup

f ∈F

|Pnf − Pf | a.s. − − → 0. Definition (P-Donsker) Gn = √n(Pn − P) converges in law to a tight limit process GP in l∞(F), also known as a P-Brownian bridge.

Marquis Hou (UCSD) Learning Proofs 10 / 16

Empirical Process in General Sample Space Measurability and P-Donsker Class

In Gin´ e and Zinn (1984), there is a long list of criteria for proper class F. Usually, we need additional measurability for uncountable F: LSM SM LDM DM NLSM NLDM

Marquis Hou (UCSD) Learning Proofs 11 / 16

Empirical Bootstrap Weak Convergence with Donsker Class

Empirical Bootstrap

In Gin´ e and Zinn (1990), a general convergence theorem for empirical Bootstrap is established. We need to assume certain measurability condition F ∈ M(P) NLDM(P) for F and NLSM(P) for F2 and F′2. Theorem (Gin´ e and Zinn 1990) Let F ∈ M(P), then the following are equivalent: (a) The envelope F for F is in L2(P) and F is P-Donsker with limit GP. (b) There exists a centered tight Gaussian process G on F such that √n(P∗

n − Pn) → G weakly in l∞(F).

If either one holds, then G = GP.

Marquis Hou (UCSD) Learning Proofs 12 / 16

Empirical Bootstrap Weak Convergence with Donsker Class

Convergence via Bounded Lipschitz Metric

The equivalence of weak convergence in l∞(F): L{Gn} L{G} ⇔ sup

h∈BL1(l∞(F))

|E∗h(Gn) − Eh(G)| → 0 where BL1 is the space of functions whose Lipschitz norm is bounded by 1. Theorem For every P-Donsker class F with envelope function F, i.e. |f (ω)| ≤ F(ω) < ∞ for all ω ∈ Ω and f ∈ F. sup

h∈BL1(l∞(F))

|EMh(G ∗

n ) − Eh(GP)| P

− → 0 Moreover, G ∗

n is asymptotically measurable. If P∗F 2 < ∞, then the

convergence is outer almost surely as well.

Marquis Hou (UCSD) Learning Proofs 13 / 16

Empirical Bootstrap Functional δ-Method

Theorem (Delta method for Bootstrap) Let D be a normed space and let φ : Dφ ⊂ D → Rk be Hadamard differentiable at θ tangentially to a subspace D0. Let ˆ θn and ˆ θ∗ be maps with values in Dφ such that √n(ˆ θn − θ) L − → T, tight in D0. suph∈BL1(D) |EMh(√n(ˆ θ∗

n − ˆ

θ)) − Eh(T)| P − → 0. Then suph∈BL1(D) |EMh(√n(φ(ˆ θ∗

n) − φ(ˆ

θ))) − Eh(φ′

θ(T))| P

− → 0.

Marquis Hou (UCSD) Learning Proofs 14 / 16

Empirical Bootstrap Functional δ-Method

An Application

Corollary (Empirical distribution function) The class F = {ft : ft = 1(−∞,t]} is Donsker, so the empirical distribution function Fn satisfies the condition for the preceding theorem. Thus, conditionally on sample, √n(φ(F ∗

n ) − φ(Fn)) converges in distribution to

the same limit as √n(φ(Fn) − φ(F)), for every Hadamard-differentiable function φ, e.g. quantiles and trimmed-means.

Marquis Hou (UCSD) Learning Proofs 15 / 16

Empirical Bootstrap Functional δ-Method

The End

Marquis Hou (UCSD) Learning Proofs 16 / 16