STAT 113 Confidence Intervals Colin Reimer Dawson Oberlin College - - PowerPoint PPT Presentation

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

STAT 113 Confidence Intervals

Colin Reimer Dawson

Oberlin College

October 3, 2017 1 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Outline

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals Bootstrap Resampling Bootstrap Confidence Intervals Bootstrap Percentile Intervals 2 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Two Main Goals of Inference

• 1. Assessing strength of evidence about “yes/no” questions

(hypothesis testing)

• 2. Estimating unknown quantities in a population using a sample

(confidence intervals) 3 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Statistics vs. Parameters

• Summary values (like mean, median, standard deviation) can

be computed for populations or for samples.

• In a population, such a summary value is called a parameter
• In a sample, these values are called statistics, and are used to

estimate the corresponding parameter Value Population Parameter Sample Statistic Mean µ ¯ X Proportion p ˆ p Correlation ρ r Slope of a Line β1 ˆ β1 Difference in Means µ1 − µ2 ¯ X1 − ¯ X2 . . . . . . . . . 4 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Using Samples to Make Estimates About Populations

• The set of all gumballs from my factory is my population.
• The mean flavor-life in the population is a population

parameter (write µ for the pop. mean)

• Ideally I can test a random sample
• The mean flavor-life in the sample is a sample statistic (write

¯ x for the sample mean).

Statistic : Sample :: Parameter : Population

5 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Variability due to Sampling

• Samples are imperfect reflections of the population.
• However, some populations are more compatible with the

sample than others.

• If we imagine a continuum of populations (or just population

means), some are more plausible than others because they make the data more likely. 6 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Sampling Distributions

Sampling Distribution Definition

Consider all possible random samples of a fixed size, n from a

• population. Each one has its own value for a particular statistic

(like ¯ x). A sampling distribution is the collection of all of of those ¯ x values (or whatever the statistic is) 7 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Sampling Distribution of Gumball Means

Population flavor−life (min.) Density 60 65 70 75 0.00 0.15 mean = 66.77 Sample Mean Flavor Life (n = 10) Density 60 65 70 75 0.0 0.4 s = 0.9

8 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Self-Check Quiz

• 1. What are the cases in the context of a sampling distribution?

Possible samples of a fixed size n

• 2. What is the variable in the relevant sampling distribution for

the gumball life example? Each case has its own sample mean 9 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Standard Error

Standard Error Definition

The distribution of a quantitative variable has a standard deviation. The sampling distribution of a quantitative sample statistic (like a mean) has a standard deviation too. This has a special name: the standard error (e.g., “of the mean”). 10 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Sampling Distribution of Gumball Means

Population flavor−life (min.) Density 60 65 70 75 0.00 0.15 mean = 66.77 Sample Mean Flavor Life (n = 10) Density 60 65 70 75 0.0 0.4 s = 0.9

11 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Properties of Sampling Distribution

Most (about 95%) of simple random samples have a sample mean (¯ x) which is within 2 Standard Errors of the population mean (µ).

12 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Sampling Distribution of Gumball Means

Population flavor−life (min.) Density 60 65 70 75 0.00 0.15 mean = 66.77 Sample Mean Flavor Life (n = 10) Density 60 65 70 75 0.0 0.4 s = 0.9

13 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Properties of Sampling Distribution

Most (about 95%) of simple random samples have a sample mean (¯ x) which is within 2 Standard Errors of the population mean (µ). The population mean µ is within 2 Standard Errors of most (about 95%) sample means.

Deeeeeep.... 14 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Outline

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals Bootstrap Resampling Bootstrap Confidence Intervals Bootstrap Percentile Intervals 15 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Margins of Error

In a Gallup poll released yesterday, a sample of 1500 adults in the U.S. voters were asked whether they approved or disapproved of the job that Donald Trump is doing as president. 42% of respondents said “approve” and 54% said disapprove. The poll’s margin of error was 3 percentage points.

• What’s the meaning of that 3%?

Margin of Error

It defines a range of “plausible” values for each population

• proportion. Precisely, a 95% margin of error of 3 points means that

95% of surveys with the same procedure and sample size will yield sample statistics which are within 3 points of the corresponding population parameter. 16 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Confidence Intervals

• A point estimate of some population parameter (like a mean),

together with some measure of our confidence/uncertainty (e.g., MoE), defines a confidence interval.

• Can be written in the form “statistic ± MoE”.

Stating Confidence Intervals

• “With 95% confidence, the mean flavor-life of our gumballs is

between 65.3 and 67.1 minutes.”

• “With 95% confidence, between 43 (i.e, 46 − 3) and 49 (i.e.,

46 + 3) percent of registered voters prefer Hillary Clinton to Donald Trump.”

• “With 95% confidence, between 39 (42 − 3) and 45 (42 + 3)

percent of U.S. adults approve of the president’s job performance.” 17 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Self-Check: Confidence Intervals

HBO Sports/Marist gave 1253 U.S. adults the following poll question in spring 2015 (I have edited for length): "Top college men’s football and basketball programs bring in a lot of money to their schools... Do you think student athletes in [these top programs] should be paid for the hours they are required to spend practicing, traveling, and playing on the team, OR should not be paid given the value of their scholarship and a chance to earn a degree?" This poll’s 95% margin of error is 2.8%. The results are given in the following table. Find a 95% confidence interval for the percentage of U.S. adults who chose the first option. Should be paid Should not be paid Unsure 33% 65% 2% 18 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

How to Determine the Margin of Error?

The population mean µ is within 2 Standard Errors of most (about 95%) sample means (from simple random samples). Margin of Error

A 95% margin of error of 3 points means that 95% of surveys with the same procedure and sample size will yield sample statistics which are within 3 points of the corresponding population parameter.

If the sampling distribution is approximately Normal (bell-shaped), then 95% Margin of Error is about 2 Standard Errors.

19 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Confidence Interval of Gumball Flavor Life

Sample Mean Flavor Life (n = 10) Density 60 65 70 75 0.0 0.4 s = 0.9 60 65 70 75 Sample flavor−life (min.)

• ●
• mean = 67.23

20 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

A Different Confidence Interval of Gumball Flavor Life

Sample Mean Flavor Life (n = 10) Density 60 65 70 75 0.0 0.4 s = 0.9 60 65 70 75 Sample flavor−life (min.)

• ●
• ●●
• mean = 71.08

21 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Example: Carbon in Forest Biomass

• Scientists hoping to curb deforestation estimate1 that the

carbon stored in tropical forests in Latin America, sub-Saharan Africa, and southeast Asia has a total biomass of 247 gigatons.

• To arrive at this estimate, they first estimate the mean

amount of carbon per square kilometer.

• Based on a sample of size n = 4079 inventory plots, the

sample mean is ¯ x = 11600 tons with a standard error of 1000 tons.

• Give and interpret a 95% confidence interval for the carbon

per km in the entire set of forests.

1Saatchi, S.S. et. al. “Benchmark Map of Forest Carbon Stocks in Tropical

Regions Across Three Continents,” Proceedings of the National Academy of Sciences, 5/31/11.

22 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Common Misinterpretations

• 95% CIs contain 95% of the cases in the population. False.

They represent uncertainty about a population parameter, not about individual points.

• There is a 95% chance that the sample mean falls in the 95%
• CI. False. Any given CI is centered around the sample mean for

that sample, so the sample mean is inside 100% of the time. 23 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Correct Interpretations

• 95% of samples produce confidence intervals that contain the

population parameter. True: This is the definition of a confidence interval 24 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

25 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Correct or Incorrect?

A 98% confidence interval for mean pulse rate in the Oberlin student population is 65 to 71. The interpretation “I am 98% sure that all students will have pulse rates between 65 and 71.” is

• A. Correct
• B. Incorrect Incorrect

26 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Correct or Incorrect?

A 98% confidence interval for mean pulse rate in the Oberlin student population is 65 to 71. The interpretation “I am 98% sure that the mean pulse rate for this sample of students will fall between 65 and 71” is

• A. Correct
• B. Incorrect Incorrect

27 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Correct or Incorrect?

A 98% confidence interval for mean pulse rate in the Oberlin student population is 65 to 71. The interpretation “I am 98% sure that the mean pulse rate for the population of all students will fall between 65 and 71” is

• A. Correct Correct
• B. Incorrect

28 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Correct or Incorrect?

A 98% confidence interval for mean pulse rate in the Oberlin student population is 65 to 71. The interpretation “98% of the pulse rates for students at this college will fall between 65 and 71” is

• A. Correct
• B. Incorrect Incorrect

29 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

30 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Summary

To create a 95% confidence interval for a parameter:

• 1. Take many random samples from the population, and compute

the sample statistic for each sample

• 2. Compute the standard error as the standard deviation of all

these statistics

• 3. For your actual sample, use statistic ± 2SE

31 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Ok, but...

In reality we only have one sample. How do we know what the standard error is?

• Standard error depends on population characteristics,

particularly variability

• We can use the sample to estimate not only the parameter of

interest (e.g., mean, proportion), but also the variability.

• Two approaches: (1) Simulation, (2) Probability theory

32 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Outline

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals Bootstrap Resampling Bootstrap Confidence Intervals Bootstrap Percentile Intervals 33 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Outline

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals Bootstrap Resampling Bootstrap Confidence Intervals Bootstrap Percentile Intervals 34 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Estimating the Margin of Error from One Sample

• Since we only have one sample, we have to estimate the

Margin of Error using only the information it contains.

• Idea: Let the whole sample (not just the statistic of interest)

serve as an estimate for the whole population 35 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Note: We do not literally make copies of the data, or increase our sample size, by bootstrapping! 36 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Sampling from the Pseudo-Population

• Sampling from the estimated population is equivalent to

sampling from the sample, but never “using up” the cases.

• In other words, we sample with replacement from the sample.
• The resulting sample is called a bootstrap sample.

37 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

38 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Bootstrap Statistic and Bootstrap Distribution

• We compute the relevant statistic (e.g., mean) on the

bootstrap sample. This is a bootstrap statistic.

• Over many bootstrap samples, each contributing a bootstrap

statistic, we get a bootstrap distribution.

• Each bootstrap statistic differs from the “pseudopopulation

parameter” (which is really the real sample statistic).

• We hope these differences are similar in size to the differences

between true sample statistics and population parameter.

Bootstrap statistic : Actual sample statistic :: Actual sample statistic : Actual Population Parameter

39 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

40 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Example: Daily Text Messages Count how many text messages you received yesterday (midnight to midnight), then go to http://colindawson.net/stat113/ and click on the poll link to enter the value.

41 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Population vs. Sample vs. Sampling Dist. vs. Bootstrap Dist.

Population <- read.file("http://colindawson.net/data/ames.csv") Sample <- sample(Population, size = 50) SamplingDist <- do(5000) * sample(Population, size = 50) %>% mean(~Price, data = .) BootstrapDist <- do(5000) * resample(Sample) %>% mean(~Price, data = .)

42 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Population vs. Sample vs. Sampling Dist. vs. Bootstrap Dist.

Price

• Pop. Cases

20 40 60 80 100 150000 200000

Price

• Samp. Cases

0.0 0.5 1.0 1.5 2.0 150000 200000

Mean Price Samples

200 400 600 800 150000 200000

Mean Price

• Boot. Samples

200 400 600 150000 200000

• What is the center of the

sampling distribution?

• What is the center of the

bootstrap distribution?

• How does the spread

compare? 43 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Outline

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals Bootstrap Resampling Bootstrap Confidence Intervals Bootstrap Percentile Intervals 44 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Estimating the Margin of Error

Mean Price Samples 100000 150000 200000 250000 400

• 95%

Mean Price

• Boot. Samples

100000 150000 200000 250000 600

• 95%
• The spread of the bootstrap distribution approximates the

spread of the true sampling distribution.

• We can use the bootstrap distribution to get a Margin of Error

for our Confidence Interval

• Where should the center of the CI be?

45 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Outline

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals Bootstrap Resampling Bootstrap Confidence Intervals Bootstrap Percentile Intervals 46 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Adjusting the Confidence Level

If the sampling distribution is approximately Normal, then a 95% Margin of Error is about 2 Standard Errors. If the bootstrap distribution is approximately Normal, 95% of the bootstrap statistics are within 2 SE of the boostrap center (i.e.,

• riginal sample stat.). That is, 95% of bootstrap statistics are

within the 95% CI. If the bootstrap distribution is symmetric, then capturing the middle X% of the bootstrap statistics yields an X% confidence interval! 47 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Estimating the Margin of Error

Mean Price Samples 100000 150000 200000 250000 400

• 99%

Mean Price

• Boot. Samples

100000 150000 200000 250000 600

• 99%
• If we want a 99% CI, we need a MoE such that 99% of sample

stats are within that MoE of the population parameter.

• Since the bootstrap dist. has similar spread to the true

sampling dist., we can estimate such an MoE there

• Then build a CI around the sample stat. (aka center of

boostrap dist.) with that MoE. 48 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

CI with Arbitrary Confidence Level

### 99% CI goes from 0.5 percentile to 99.5 percentile of bootstrap dist. CI <- quantile(~result, data = BootstrapDist, probs = c(0.005, 0.995)) CI 0.5% 99.5% 159880.1 207599.8

Bootstrap Mean Density

0.00000 0.00001 0.00002 0.00003 0.00004 0.00005 160000 180000 200000 220000

49 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Example: Atlanta Commutes http://lock5stat.com/StatKey

50 / 51

Sampling Distributions Confidence Intervals Bootstrap Confidence Intervals

Summary: Bootstrap CIs

To generate a bootstrap distribution, we

• 1. Generate bootstrap samples by sampling with replacement

from the original sample, using the same sample size

• 2. Compute the statistic of interest, a bootstrap statistic, for

each of the bootstrap samples

• 3. Collect the statistics for many bootstrap samples to form a

bootstrap distribution If the bootstrap distribution is symmetric, an X% CI can be estimated by taking the range of the middle X% of the bootstrap statistics. 51 / 51