Chapter 5 Limit Theorems Peng-Hua Wang Graduate Institute of - - PowerPoint PPT Presentation

▶
chapter 5
SMART_READER_LITE
LIVE PREVIEW

Chapter 5 Limit Theorems Peng-Hua Wang Graduate Institute of - - PowerPoint PPT Presentation

May 13, 2023 30 likes •182 views

Chapter 5 Limit Theorems Peng-Hua Wang Graduate Institute of Communication Engineering National Taipei University Chapter Contents 5.1 Some Useful Inequalities 5.2 The Weak Law of Large Numbers 5.3 Convergence in Probability 5.4 The Central

slide-1
SLIDE 1

Chapter 5

Limit Theorems

Peng-Hua Wang

Graduate Institute of Communication Engineering National Taipei University

slide-2
SLIDE 2

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 2/15

Chapter Contents

5.1 Some Useful Inequalities 5.2 The Weak Law of Large Numbers 5.3 Convergence in Probability 5.4 The Central Limit Theorem 5.5 The Strong Law of Large Numbers

slide-3
SLIDE 3

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 3/15

5.1 Some Useful Inequalities

slide-4
SLIDE 4

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 4/15

Markov Inequality

If a random variable X ≥ 0, then P(X ≥ a) ≤ E[X] a , for all a > 0 Proof. E[X] =

∞

x fX(x)dx ≥

∞

a

x fX(x)dx

≥

∞

a

a fX(x)dx = a

∞

a

fX(x)dx

= aP(X ≥ a)

slide-5
SLIDE 5

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 5/15

Markov Inequality

■ Relationship between probability and mean. Use mean

to estimate probability.

■ Example 5.1. X is uniformly distributed in (0, 4).

E[X] = 2.

◆ P(X > 2) ≤ 2/2 = 1, P(X > 2) = 1/2 ◆ P(X > 3) ≤ 2/3, P(X > 3) = 1/4 ◆ The bounds can be very loose.

slide-6
SLIDE 6

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 6/15

Chebyshev Inequality

If a random variable X has mean µ and variance σ2, then P(|X − µ| ≥ c) ≤ σ2 c2 , for all c > 0 Proof. P(|X − µ| ≥ c) = P((X − µ)2 ≥ c2)

≤ E[(X − µ)2]

c2

= σ2

c2

  • Remark.

P(|X − µ| ≥ kσ) ≤ 1 k2

slide-7
SLIDE 7

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 7/15

Chebyshev Inequality

■ Relationship between probability and mean and

  • variance. Use mean and variance to estimate probability.

■ Example 5.2. X is uniformly distributed in (0, 4).

E[X] = 2 and Var(X) = 16/12

◆ P(|X − 2| > 1) ≤ 16/12, ◆ The bounds can be very loose.

slide-8
SLIDE 8

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 8/15

5.2 The Weak Law Of Large Numbers

slide-9
SLIDE 9

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 9/15

Inequality for Sample Mean

■ Let X1, X2, ... be a sequence of independent identically

distributed random variables with mean µ and variance σ2. Let Mn be the sample mean: Mn = X1 + X2 + · · · + Xn n

■ We have E[Mn] = µ and Var(Mn) = σ2/n. ■ Apply Chebyshev inequality. For any ǫ > 0 we have

P(|Mn − µ| ≥ ǫ) ≤ σ2 nǫ2

■ The right-hand side of this inequality goes to zero as n

increases.

slide-10
SLIDE 10

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 10/15

The Weak Law of Large Numbers

■ Therefore, sample mean (a random variable) approaches

the mean in probability as n approaches infinity.

■ The Weak Law of Large Numbers

P

  • X1 + X2 + · · · + Xn

n

− µ

  • ≥ ǫ
  • → 0
slide-11
SLIDE 11

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 11/15

5.4 The Central Limit Theorem

slide-12
SLIDE 12

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 12/15

The Central Limit Theorem

■ Let X1, X2, ... be a sequence of independent identically

distributed random variables with mean µ and variance σ2. Define Zn as: Zn = X1 + X2 + · · · + Xn − nµ σ√n

■ It can be shown that the CDF of Yn converges to the CDF

  • f standard normal distribution.

lim

n→∞ P(Zn ≤ z) = Φ(z) =

z

−∞

1

√

2π e−x2/2dx

slide-13
SLIDE 13

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 13/15

Example 5.12

Toss a fair coin for 36 times. Estimate the probability that the head occurs for less than ro equal to 21 times. Let Xi be iid Bernoulli rv’s with parameter 0.5. We have µ = 0.5 and σ2 = 0.5(1 − 0.5). The probability of interest is P (X1 + X2 + · · · + X36 ≤ 21)

=P

X1 + X2 + · · · + X36 − 36µ

√

36σ

≤ 21 − 36µ √

36σ

  • ≈ Φ(1) ≈ 0.8413

The exact value is

21

∑

n=0

36 n

  • (0.5)36 = 0.8785
slide-14
SLIDE 14

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 14/15

Binomial Distribution and CLT

■ Let Xi be iid Bernoulli rv’s with parameter p. Let

Sn = X1 + X2 + · · · Xn. We know that Sn be a binomial rv with parameter n and p.

■ The probability of P(k ≤ Sn ≤ ℓ) is

P(k ≤ Sn ≤ ℓ) =

ℓ

∑

i=k

n i

  • pi(1 − p)n−i

■ By CLT, we have

P(k ≤ Sn ≤ ℓ) = P

  • k − np
  • np(1 − p) ≤

Sn − np

  • np(1 − p) ≤

ℓ − np

  • np(1 − p)
  • ≈ Φ
  • ℓ − np
  • np(1 − p)
  • − Φ
  • k − np
  • np(1 − p)
slide-15
SLIDE 15

Peng-Hua Wang, June 4, 2012 Probability, Chap 2 - p. 15/15

Binomial Distribution and CLT

■ A more accurate approximation can be obtained by

replacing k and ℓ by k − 1/2 and ℓ + 1/2. The is called the De Moivre-Laplace approximation. P(k ≤ Sn ≤ ℓ) ≈ Φ

  • ℓ + 1/2 − np
  • np(1 − p)
  • − Φ
  • k − 1/2 − np
  • np(1 − p)
  • ■ In Example 5.12,

P (X1 + X2 + · · · + X36 ≤ 21)

=P

X1 + X2 + · · · + X36 − 36µ

√

36σ

≤ 21 + 1/2 − 36µ √

36σ

  • ≈Φ((21.5 − 6)/3) ≈ 0.879