Gov 2000: 4. Sums, Means, and Limit Theorems Matthew Blackwell - - PowerPoint PPT Presentation

Gov 2000: 4. Sums, Means, and Limit Theorems

Matthew Blackwell

Fall 2016

1 / 60

• 1. Sums and Means of Random Variables
• 2. Useful Inequalities
• 3. Law of Large Numbers
• 4. Central Limit Theorem
• 5. More Exotic CLTs*
• 6. Wrap-up

2 / 60

Where are we? Where are we going?

• Probability: formal way to quantify uncertain
• utcomes/random variables.
• Last week: how to work with multiple r.v.s at the same time.
• This week: applying those ideas to study large random

samples

3 / 60

Large random samples

• In real data, we will have a set of 𝑜 measurements on a

variable: 𝑌1, 𝑌2, … , 𝑌𝑜

• Or we might have a set of 𝑜 measurements on two variables:

(𝑌1, 𝑍1), (𝑌2, 𝑍2), … , (𝑌𝑜, 𝑍𝑜)

• Empirical analyses: sums or means of these 𝑜 measurements

▶ Almost all statistical procedures involve a sum/mean. ▶ What are the properties of these sums and means? ▶ Can they tell us anything about the distribution of 𝑌𝑗?

• Asymptotics: what can we learn as 𝑜 gets big?

4 / 60

1/ Sums and Means of Random Variables

5 / 60

Sums and means are random variables

• If 𝑌1 and 𝑌2 are r.v.s, then 𝑌1 + 𝑌2 is a r.v.

▶ Has a mean 𝔽[𝑌1 + 𝑌2] and a variance 𝕎[𝑌1 + 𝑌2]

• The sample mean is a function of sums and so it is a r.v. too:

̅ 𝑌 = 𝑌1 + 𝑌2 2

6 / 60

Distribution of sums/means

𝑌1 𝑌2 𝑌1 + 𝑌2 ̅ 𝑌 draw 1 20 71 91 45.5 draw 2 12 66 78 39 draw 3 59 75 134 67 draw 4 3 58 61 30.5 ⋮ ⋮ ⋮ ⋮ ⋮ distribution

• f the sum

distribution

• f the mean

7 / 60

Independent and identical r.v.s

• We often will work with independent and identically

distributed r.v.s, 𝑌1, … , 𝑌𝑜

▶ Random sample of 𝑜 respondents on a survey question. ▶ Written “i.i.d.”

• Independent: 𝑌𝑗 ⟂

⟂ 𝑌𝑘 for all 𝑗 ≠ 𝑘

• Identically distributed: 𝑔𝑌𝑗(𝑦) is the same for all 𝑗

▶ 𝔽[𝑌𝑗] = 𝜈 for all 𝑗 ▶ 𝕎[𝑌𝑗] = 𝜏2 for all 𝑗 8 / 60

Distribution of the sample mean

• Sample mean of i.i.d. r.v.s: 𝑌𝑜 = 1

𝑜 ∑𝑜 𝑗=1 𝑌𝑗

• 𝑌𝑜 is a random variable, what is its distribution?

▶ What is the expectation of this distribution, 𝔽[𝑌𝑜]? ▶ What is the variance of this distribution, 𝕎[𝑌𝑜]? ▶ What is the p.d.f. of the distribution?

• How do they relate to the expectation, variance of 𝑌1, … , 𝑌𝑜?

9 / 60

Properties of the sample mean

Mean and variance of the sample mean

Suppose that 𝑌1, … , 𝑌𝑜 is are i.i.d. r.v.s with 𝔽[𝑌𝑗] = 𝜈 and 𝕎[𝑌𝑗] = 𝜏2. Then: 𝔽[𝑌𝑜] = 𝜈 𝕎[𝑌𝑜] = 𝜏2 𝑜

• Key insights:

▶ Sample mean get the right answer on average ▶ Variance of 𝑌𝑜 depends on the variance of 𝑌𝑗 and the sample

size

▶ Not dependent on the (full) distribution of 𝑌𝑗!

• Standard error of the sample mean: √𝕎[𝑌𝑜] = 𝜏

√𝑜

• You’ll prove both of these facts in this week’s HW.

10 / 60

2/ Useful Inequalities

11 / 60

Why inequalities?

• Behavior of r.v.s depend on their distribution, but we often

don’t know (or don’t want to assume) a distribution.

• Today, we’ll discuss results for r.v.s with any distribution

subject to some restrictions like fjnite variance.

• Why study these?

▶ Build toward massively important results like LLN ▶ Inequalities used regularly throughout statistics ▶ Gives us some practice with proofs/analytic reasoning 12 / 60

Markov Inequality

Markov Inequality

Suppose that 𝑌 is r.v. such that ℙ(𝑌 ≥ 0) = 1. Then, for every real number 𝑢 > 0, ℙ(𝑌 ≥ 𝑢) ≤ 𝔽[𝑌] 𝑢 .

• For instance, if we know that 𝔽[𝑌] = 1, then

ℙ(𝑌 ≥ 100) ≤ 0.01

• Once we know the mean of a r.v., it limits how much

probability can be in the tail.

13 / 60

Markov Inequality Proof

• For discrete 𝑌:

𝔽[𝑌] = ∑

𝑦

𝑦𝑔𝑌(𝑦) = ∑

𝑦<𝑢

𝑦𝑔𝑌(𝑦) + ∑

𝑦≥𝑢

𝑦𝑔𝑌(𝑦)

• Because 𝑌 is nonnegative, 𝔽[𝑌] ≥ ∑𝑦≥𝑢 𝑦𝑔𝑌(𝑦)
• Since 𝑦 ≥ 𝑢, then ∑𝑦≥𝑢 𝑦𝑔𝑌(𝑦) ≥ ∑𝑦≥𝑢 𝑢𝑔𝑌(𝑦)
• But this is just ∑𝑦≥𝑢 𝑢𝑔𝑌(𝑦) = 𝑢 ∑𝑦≥𝑢 𝑔𝑌(𝑦) = 𝑢ℙ(𝑌 ≥ 𝑢)
• Implies 𝔽[𝑌] ≥ 𝑢ℙ(𝑌 ≥ 𝑢)

14 / 60

Chebyshev Inequality

Chebyshev Inequality

Suppose that 𝑌 is r.v. for which 𝕎[𝑌] < ∞. Then, for every real number 𝑢 > 0, ℙ(|𝑌 − 𝔽[𝑌]| ≥ 𝑢) ≤ 𝕎[𝑌] 𝑢2 .

• The variance places limits on how far an observation can be

from its mean.

15 / 60

Proof of Chebyshev

• Let 𝑍 = (𝑌 − 𝔽[𝑌])2

▶ ⇝ ℙ(𝑍 ≥ 0) = 1 (nonnegative) ▶ 𝔽[𝑍] = 𝔽[(𝑌 − 𝔽[𝑌])2] = 𝕎[𝑌] (defjnition of variance)

• Note that if |𝑌 − 𝔽[𝑌]| ≥ 𝑢 then 𝑍 ≥ 𝑢2 because we just

squared both sides.

• Thus, ℙ(|𝑌 − 𝔽[𝑌]| ≥ 𝑢) = ℙ(𝑍 ≥ 𝑢2)
• Apply Markov’s inequality:

ℙ(|𝑌 − 𝔽[𝑌]| ≥ 𝑢) = ℙ(𝑍 ≥ 𝑢2) ≤ 𝔽[𝑍] 𝑢2 = 𝕎[𝑌] 𝑢2

16 / 60

Application: planning a survey

• Suppose we want to estimate the proportion of voters who will

vote for Donald Trump, 𝑞, from a random sample of size 𝑜.

▶ 𝑌1, 𝑌2, … , 𝑌𝑜 indicating voting intention for Trump for each

respondent.

▶ By our earlier, calculation, 𝔽[𝑌𝑜] = 𝑞 and 𝕎[𝑌𝑜] = 𝜏2

𝑜

▶ Since this is a Bernoulli r.v., we have 𝜏2 = 𝑞(1 − 𝑞)

• What does 𝑜 need to be to have at least 0.95 probability that

𝑌𝑜 is within 0.02 of the true 𝑞?

▶ How to guarantee a margin of error of ± 2 percentage points? 17 / 60

Application: planning a survey

• What does 𝑜 have to be so that

ℙ(|𝑌𝑜 − 𝑞| ≤ 0.02) ≥ 0.95 ⟺ ℙ(|𝑌𝑜 − 𝑞| ≥ 0.02) ≤ 0.05

• Applying Chebyshev:

ℙ(|𝑌𝑜 − 𝑞| ≥ 0.02) ≤ 𝕎[𝑌𝑜] 0.022 = 𝑞(1 − 𝑞) 0.0004𝑜

• We don’t know 𝕎[𝑌𝑗] = 𝑞(1 − 𝑞), but:

▶ Conservative to use largest possible variance. ▶ It can’t be bigger than 𝑞(1 − 𝑞) ≤ (1/2) ⋅ (1/2) = (1/4)

ℙ(|𝑌𝑜 − 𝑞| ≥ 0.02) ≤ 𝑞(1 − 𝑞) 0.0004𝑜 ≤ 1 0.0016𝑜

• We want this probability to be bounded by 0.05 so we need

(1/0.0016𝑜) ≤ 0.05, which gives us 𝑜 ≥ 12, 500!!

18 / 60

Application: planning a survey

• Do we really need 𝑜 ≥ 12, 500 to get a margin of error of ±2

percentage points?

• No! Chebyshev provides a bound that is guaranteed to hold,

but actual probabilities are much smaller.

▶ We’re also using the “worst-case” variance of 0.25.

• Let’s simulate 1000 samples of size 𝑜 = 12500 with 𝑞 = 0.4

and show the distribution of the means.

▶ What proportion of these are within 0.02 of 𝑞? 19 / 60

Application: planning a survey

nsims <- 1000 holder <- rep(NA, times = nsims) for (i in 1:nsims) { this.samp <- rbinom(n = 12500, size = 1, prob = 0.4) holder[i] <- mean(this.samp) } mean(abs(holder - 0.4) > 0.02) ## [1] 0

• 0.03
• 0.02
• 0.01

0.00 0.01 0.02 0.03 20 40 60 80 xn − p Density

20 / 60

3/ Law of Large Numbers

21 / 60

Current knowledge

• For i.i.d. r.v.s, 𝑌1, … , 𝑌𝑜, with 𝔽[𝑌𝑗] = 𝜈 and 𝕎[𝑌𝑗] = 𝜏2 we

know that:

▶ Expectation is 𝔽[𝑌𝑜] = 𝔽[𝑌𝑗] = 𝜈 ▶ Variance is 𝕎[𝑌𝑜] = 𝜏2

𝑜 where 𝜏2 = 𝕎[𝑌𝑗]

▶ Some bounds on tail probabilities from Chebyshev. ▶ None of these rely on a specifjc distribution for 𝑌𝑗!

• Can we say more about the distribution of the sample mean?
• Yes, but we need to think about how 𝑌𝑜 changes as 𝑜 gets big.

22 / 60

Sequence of sample means

• What can we say about the sample mean 𝑜 gets large?
• Need to think about sequences of sample means with

increasing 𝑜: 𝑌1 = 𝑌1 𝑌2 = (1/2) ⋅ (𝑌1 + 𝑌2) 𝑌3 = (1/3) ⋅ (𝑌1 + 𝑌2 + 𝑌3) 𝑌4 = (1/4) ⋅ (𝑌1 + 𝑌2 + 𝑌3 + 𝑌4) 𝑌5 = (1/5) ⋅ (𝑌1 + 𝑌2 + 𝑌3 + 𝑌4 + 𝑌5) ⋮ 𝑌𝑜 = (1/𝑜) ⋅ (𝑌1 + 𝑌2 + 𝑌3 + 𝑌4 + 𝑌5 + ⋯ + 𝑌𝑜)

• Note: this is a sequence of random variables!

23 / 60

Convergence in Probability

Convergence in probability

A sequence of random variables, 𝑎1, 𝑎2, …, is said to converge in probability to a value 𝑐 if for every 𝜁 > 0, ℙ(|𝑎𝑜 − 𝑐| > 𝜁) → 0, as 𝑜 → ∞. We write this 𝑎𝑜

𝑞

→ 𝑐.

• Basically: probability that 𝑎𝑜 lies outside any (teeny, tiny)

interval around 𝑐 approaches 0 as 𝑜 → ∞

• Wooldridge writes plim(𝑎𝑜) = 𝑐 if 𝑎𝑜

𝑞

→ 𝑐.

24 / 60

Law of large numbers

Theorem: Weak Law of Large Numbers

Let 𝑌1, … , 𝑌𝑜 be a an i.i.d. draws from a distribution with mean 𝜈 and fjnite variance 𝜏2. Let 𝑌𝑜 = 1

𝑜 ∑𝑜 𝑗=1 𝑌𝑗. Then, 𝑌𝑜 𝑞

→ 𝜈.

• Intuition: The probability of 𝑌𝑜 being “far away” from 𝜈 goes

to 0 as 𝑜 gets big.

▶ The distribution of 𝑌𝑜 “collapses” on 𝜈

• No assumptions about the distribution of 𝑌𝑗 beyond i.i.d. and

a fjnite variance!

25 / 60

LLN proof

• Proof: by Chebyshev and properties of probabilities, we have

0 ≤ ℙ(|𝑌𝑜 − 𝜈| ≥ 𝜁) ≤ 𝕎[𝑌𝑜] 𝜁2 = 𝜏2 𝑜𝜁2

• As 𝑜 → ∞, we know that 𝜏2/𝑜𝜁2 → 0 which by the sandwich

theorem implies lim

𝑜→∞ ℙ(|𝑌𝑜 − 𝜈| > 𝜁) = 0

26 / 60

LLN by simulation in R

• Draw difgerent sample sizes from Exponential distribution with

rate 0.5

• ⇝ 𝔽[𝑌𝑗] = 2

nsims <- 10000 holder <- matrix(NA, nrow = nsims, ncol = 6) for (i in 1:nsims) { s5 <- rexp(n = 5, rate = 0.5) s15 <- rexp(n = 15, rate = 0.5) s30 <- rexp(n = 30, rate = 0.5) s100 <- rexp(n = 100, rate = 0.5) s1000 <- rexp(n = 1000, rate = 0.5) s10000 <- rexp(n = 10000, rate = 0.5) holder[i, 1] <- mean(s5) holder[i, 2] <- mean(s15) holder[i, 3] <- mean(s30) holder[i, 4] <- mean(s100) holder[i, 5] <- mean(s1000) holder[i, 6] <- mean(s10000) }

27 / 60

LLN in action

1 2 3 4 1 2 3 4 5 6 Density n = 15

• Distribution of 𝑌15

28 / 60

LLN in action

1 2 3 4 1 2 3 4 5 6 Density n = 30

• Distribution of 𝑌30

29 / 60

LLN in action

1 2 3 4 1 2 3 4 5 6 Density n = 100

• Distribution of 𝑌100

30 / 60

LLN in action

1 2 3 4 1 2 3 4 5 6 Density n = 1000

• Distribution of 𝑌1000

31 / 60

Properties of convergence in probability

• 1. if 𝑌𝑜

𝑞

→ 𝑑, then 𝑕(𝑌𝑜)

𝑞

→ 𝑕(𝑑) for any continuous function 𝑕.

• 2. if 𝑌𝑜

𝑞

→ 𝑏 and 𝑎𝑜

𝑞

→ 𝑐, then

▶ 𝑌𝑜 + 𝑎𝑜

𝑞

→ 𝑏 + 𝑐

▶ 𝑌𝑜𝑎𝑜

𝑞

→ 𝑏𝑐

▶ 𝑌𝑜/𝑎𝑜

𝑞

→ 𝑏/𝑐 if 𝑐 > 0

• Thus, by LLN:

▶ (𝑌𝑜)

2 𝑞

→ 𝜈2

▶ log(𝑌𝑜)

𝑞

→ log(𝜈)

32 / 60

4/ Central Limit Theorem

33 / 60

Current knowledge

• For i.i.d. r.v.s, 𝑌1, … , 𝑌𝑜, with 𝔽[𝑌𝑗] = 𝜈 and 𝕎[𝑌𝑗] = 𝜏2 we

know that:

▶ 𝔽[𝑌𝑜] = 𝜈 and 𝕎[𝑌𝑜] = 𝜏2

𝑜

▶ 𝑌𝑜 converges to 𝜈 as 𝑜 gets big ▶ Chebyshev provides some bounds on probabilities. ▶ Still no distributional assumptions about 𝑌𝑗!

• Can we say more?

▶ Can we approximate Pr(𝑏 < 𝑌𝑜 < 𝑐)? ▶ What family of distributions (Binomial, Uniform, Gamma,

etc)?

• Again, need to analyze when 𝑜 is large.

34 / 60

Convergence in Distribution

Convergence in distribution

Let 𝑎1, 𝑎2, …, be a sequence of r.v.s, and for 𝑜 = 1, 2, … let 𝐺𝑜(𝑨) be the c.d.f. of 𝑎𝑜. Then it is said that 𝑎1, 𝑎2, … converges in distribution to r.v. 𝑋 with c.d.f. 𝐺𝑋 if lim

𝑜→∞ 𝐺𝑜(𝑦) = 𝐺𝑋(𝑦),

which we write as 𝑎𝑜

𝑒

→ 𝑋.

• Basically: when 𝑜 is big, the distribution of 𝑎𝑜 is very similar

to the distribution of 𝑋

• We use c.d.f.s here to avoid messy details with discrete vs

continuous.

• If 𝑌𝑜

𝑞

→ 𝑌, then 𝑌𝑜

𝑒

→ 𝑌

35 / 60

Standardizing an r.v.

• Common to standardize a r.v. by subtracting its expectation

and dividing by its standard deviation: 𝑎 = 𝑌 − 𝔽[𝑌] √𝕎[𝑌]

• Possible to show that for any 𝑌, we have (try to prove these

to yourself):

▶ 𝔽[𝑎] = 0 ▶ 𝕎[𝑎] = 1

• Sometimes called a z-score.

36 / 60

Central Limit Theorem

Central Limit Theorem

Let 𝑌1, … , 𝑌𝑜 be i.i.d. r.v.s from a distribution with mean 𝜈 and variance 0 < 𝜏2 < ∞. Then, 𝑌𝑜 − 𝜈 𝜏/√𝑜

𝑒

→ 𝑂(0, 1).

• Distribution free! We don’t have to make specifjc assumptions

about the distribution of 𝑌𝑗

• Implies that 𝑌𝑜 ∼ 𝑂(𝜈, 𝜏2/𝑜)

▶ ⇝ easy approximations to probability statements about 𝑌𝑜

when 𝑜 is big!

37 / 60

CLT by simulation in R

set.seed(2138) nsims <- 10000 holder2 <- matrix(NA, nrow = nsims, ncol = 6) for (i in 1:nsims) { s5 <- rbinom(n = 5, size = 1, prob = 0.25) s15 <- rbinom(n = 15, size = 1, prob = 0.25) s30 <- rbinom(n = 30, size = 1, prob = 0.25) s100 <- rbinom(n = 100, size = 1, prob = 0.25) s1000 <- rbinom(n = 1000, size = 1, prob = 0.25) s10000 <- rbinom(n = 10000, size = 1, prob = 0.25) holder2[i, 1] <- mean(s5) holder2[i, 2] <- mean(s15) holder2[i, 3] <- mean(s30) holder2[i, 4] <- mean(s100) holder2[i, 5] <- mean(s1000) holder2[i, 6] <- mean(s10000) }

38 / 60

CLT in action

• 3
• 2
• 1

1 2 3 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Density n = 5

• Distribution of 𝑌5−𝜈

𝜏/√5

39 / 60

CLT in action

• 3
• 2
• 1

1 2 3 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Density n = 15

• Distribution of 𝑌15−𝜈

𝜏/√15

40 / 60

CLT in action

• 3
• 2
• 1

1 2 3 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Density n = 30

• Distribution of 𝑌30−𝜈

𝜏/√30

41 / 60

CLT in action

• 3
• 2
• 1

1 2 3 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Density n = 100

• Distribution of 𝑌100−𝜈

𝜏/√100

42 / 60

CLT in action

• 3
• 2
• 1

1 2 3 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Density n = 10000

• Distribution of 𝑌10000−𝜈

𝜏/√10000

43 / 60

Empirical Rule for the Normal Distribution

• 4
• 2

2 4

• If 𝑎 ∼ 𝑂(0, 1), then the following are roughly true:

44 / 60

Empirical Rule for the Normal Distribution

• 4
• 2

2 4 0.68

• If 𝑎 ∼ 𝑂(0, 1), then the following are roughly true:
• Roughly 68% of the distribution of 𝑎 is between -1 and 1.

45 / 60

Empirical Rule for the Normal Distribution

• 4
• 2

2 4 0.95

• If 𝑎 ∼ 𝑂(0, 1), then the following are roughly true:
• Roughly 68% of the distribution of 𝑎 is between -1 and 1.
• Roughly 95% of the distribution of 𝑎 is between -2 and 2.

46 / 60

Empirical Rule for the Normal Distribution

• 4
• 2

2 4 0.997

• If 𝑎 ∼ 𝑂(0, 1), then the following are roughly true:
• Roughly 68% of the distribution of 𝑎 is between -1 and 1.
• Roughly 95% of the distribution of 𝑎 is between -2 and 2.
• Roughly 99.7% of the distribution of 𝑎 is between -3 and 3.

47 / 60

Simulating the empirical rule

• Actual probability of 𝑎 ∼ 𝑂(0, 1) between −2 and 2:

pnorm(2) - pnorm(-2) ## [1] 0.9545

• Simulated probability of 𝑌𝑜−𝜈

𝜏/√𝑜 between −2 and 2:

▶ 𝑜 = 15 ⇝ 0.9683 ▶ 𝑜 = 30 ⇝ 0.9666 ▶ 𝑜 = 100 ⇝ 0.9523 ▶ 𝑜 = 1000 ⇝ 0.9551 ▶ 𝑜 = 10000 ⇝ 0.9546

• Quality of the approximation depends on the underlying

distribution of the 𝑌𝑗

▶ Obviously if 𝑌𝑗 ∼ 𝑂(0, 1) it’s going to be perfect with 𝑜 = 1 48 / 60

Slustsky’s Theorem

• Let 𝑌1, 𝑌2, … converge in distribution to some r.v. 𝑌
• Let 𝑍1, 𝑍2, … converge in probability to some number, 𝑑
• Slutsky’s Theorem gives the following result:
• 1. 𝑌𝑜𝑍𝑜 converges in distribution to 𝑑𝑌
• 2. 𝑌𝑜 + 𝑍𝑜 converges in distribution to 𝑌 + 𝑑
• Extremely useful when trying to fjgure out what the

large-sample distribution of an estimator is.

49 / 60

Application: planning a survey

• Trump example: we want the the probability of being within

0.02 from the true 𝑞 to be 95%.

• ⇝ we want 𝑜 such that:

ℙ(|𝑌𝑜 − 𝑞| > 0.02) ≤ 0.05

• By the CLT, if 𝑜 is large, then

𝑌𝑜 − 𝑞 ≈ 𝑂 (0, 𝜏2/𝑜)

• We know 𝜏2 ≤ 1/4, so to be conservative:

▶ 𝑌𝑜 − 𝑞 ≈ 𝑂 (0, 1

4𝑜)

▶ Standardizing ⇝ 𝑎 = (𝑌𝑜−𝑞)

1/√4𝑜 = 2√𝑜(𝑌𝑜 − 𝑞) ≈ 𝑂(0, 1)

• Easier to work with standardized r.v.:

ℙ(|𝑌𝑜 − 𝑞| > 0.02) ≤ 0.05 ⟺ ℙ(|𝑎| > 0.02 × 2√𝑜) ≤ 0.05

50 / 60

Application: planning a survey

• We want:

ℙ(|𝑎| > 0.04√𝑜) ≤ 0.05 ℙ(𝑎 < −0.04√𝑜) + ℙ(𝑎 > 0.04√𝑜) ≤ 0.05

• The standard normal is symmetric around 0, so:

▶ Upper tail probs = lower tail probs ▶ ℙ(𝑎 < −0.04√𝑜) = ℙ(𝑎 > 0.04√𝑜)

• Allow us to simplify:

2 × ℙ(𝑎 < −0.04√𝑜) ≤ 0.05 ℙ(𝑎 < −0.04√𝑜) ≤ 0.025

• To solve for 𝑜, we need to know 𝑟 such that ℙ(𝑎 ≤ 𝑟) = 0.025

▶ Inverse of the c.d.f. called the quantile: 𝑟 = 𝐺−1(0.025) ▶ 𝑟 = 𝐺−1(𝑞) is the (smallest) value of the r.v. such that

ℙ(𝑌 ≤ 𝑟) = 𝐺(𝑟) ≥ 𝑞

51 / 60

Application: planning a survey

• We can use the qnorm() function in R:

qnorm(0.025, mean = 0, sd = 1) ## [1] -1.96

• if −0.04√𝑜 ≤ 𝑟, then ℙ (𝑎 < −0.04√𝑜) ≤ 0.025
• So, we need −0.04√𝑜 ≤ −1.96 or 𝑜 > 2401
• Much lower than the 12,500 from Chebyshev.

52 / 60

Application: planning a survey

nsims <- 1000 holder <- rep(NA, times = nsims) for (i in 1:nsims) { this.samp <- rbinom(n = 2401, size = 1, prob = 0.4) holder[i] <- mean(this.samp) } mean(abs(holder - 0.4) > 0.02) ## [1] 0.052

• 0.03
• 0.02
• 0.01

0.00 0.01 0.02 0.03 10 20 30 40 xn − p Density

53 / 60

5/ More Exotic CLTs*

54 / 60

CLT for non-iid r.v.s

• What if we don’t have i.i.d. r.v.s? Does the CLT still apply?
• Let 𝑌1, 𝑌2, … be independent (but not identically distributed)

with means 𝔽[𝑌𝑗] = 𝜈𝑗 and variances 𝕎[𝑌𝑗] = 𝜏2

𝑗 .

• Scaled and centered:

𝑍𝑜 = ∑𝑜

𝑗=1 𝑌𝑗 − ∑𝑜 𝑗=1 𝜈𝑗

(∑𝑜

𝑗=1 𝜏2 𝑗 ) 1/2

▶ No need to divide by 𝑜 because there are 𝑜 entries in the sum

∑𝑜

𝑗=1 𝜈𝑗

• Easy to show that 𝔽[𝑍𝑜] = 0 and 𝕎[𝑍𝑜] = 1. Does the CLT

apply?

55 / 60

Liapounov CLT

Liapounov CLT

Suppose that the r.v.s 𝑌1, 𝑌2, … are independent and that 𝔽[|𝑌𝑗 − 𝜈𝑗|3] < ∞ for 𝑗 = 1, 2, …. Also, suppose that lim

𝑜→∞

∑𝑜

𝑗=1 𝔽 [|𝑌𝑗 − 𝜈𝑗|3]

(∑𝑜

𝑗=1 𝜏2 𝑗 ) 3/2

= 0. Then, 𝑍𝑜 = ∑𝑜

𝑗=1 𝑌𝑗 − ∑𝑜 𝑗=1 𝜈𝑗

(∑𝑜

𝑗=1 𝜏2 𝑗 ) 1/2 𝑒

→ 𝑂(0, 1)

• Key condition: there isn’t one r.v.s in the sequence that is

“too big” that could dominate the sum

56 / 60

CLT for dependent sequences

• We have shown the CLT for i.i.d. and for independent r.v.s.

What about dependent sequences?

• CLT works for a dependent sequence 𝑌1, 𝑌2, ….

▶ What does dependent sequence mean? Cov[𝑌𝑗, 𝑌𝑘] ≠ 0

• Key condition for dependent CLT: r.v.s aren’t “too correlated”
• Overall conditions for CLT to hold: the sum/mean of many,

not too correlated, not too big r.v.s

57 / 60

6/ Wrap-up

58 / 60

Limitations of asymptotics

• These results are practically and theoretically very useful.
• But remember that they are approximations
• We don’t live in asymptopia—𝑜 is always fjnite.
• Asymptotics often give reasonable answers, but you can check

with simulations.

59 / 60

Review

• Sums and means of r.v.s are themselves r.v.s
• Learned about the distribution of the sample mean of i.i.d.

r.v.s

▶ Expectation 𝔽[𝑌𝑜] = 𝜈 ▶ Variance 𝕎[𝑌𝑜] = 𝜏2/𝑜 ▶ Converges in probability to true mean (LLN) ▶ Converges in distribution to a normal distribution (CLT)

• Ahead: generalizing these ideas to arbitrary estimators of

parameters.

60 / 60