Implementing Bootstrap Methods in R GETTING STARTED WITH - - PowerPoint PPT Presentation

CO-FOUNDER, LOONYCORN

www.loonycorn.com

Janani Ravi

GETTING STARTED WITH BOOTSTRAPPING IN R

Implementing Bootstrap Methods in R

Overview

Estimating statistics and calculating confidence intervals The Central Limit Theorem Conventional methods vs. bootstrap methods Advantages of bootstrapping techniques

Prerequisites and Course Outline

Exposure to statistics at the level of mean, median, and standard deviation Familiarity with probability distributions Familiarity with regression models Some exposure to R programming

Prerequisites

R Programming Fundamentals

Prerequisites

Introducing bootstrap methods

• Benefits and limitations

Bootstrapping for summary statistics

• Non-parametric bootstrapping
• Bayesian bootstrapping
• Smoothed bootstrapping

Bootstrapping for regression models

• Case resampling
• Residual resampling

Course Outline

Sample Statistics and Confidence Intervals

Two Questions

What is the average height of an American male? How confident are you of your answer?

Answering Two Questions

Take sample from population; estimate mean Calculate confidence intervals around estimate

What is the _____ of some population? How confident are you of your answer?

Generalizing to Any Statistic

Generalizing to Any Statistic

Take sample from population; estimate statistic Calculate confidence intervals around estimate You need answers to the same two questions

Example Statistics

Mean, mode, median, standard deviation Correlations, covariances Regression coefficients, R-square values Proportions, odds ratio

Estimating Population Statistic

Bootstrap Approach Conventional Approach Sample population

• nce; calculate

sample statistic Sample once; resample that sample with replacement

Establishing Confidence Intervals Around Estimate

Once the estimate has been obtained from the sample… …Need to answer the second question Need to establish confidence intervals around the estimate

Establishing Confidence Intervals

Bootstrap Approach Conventional Approach Sample once; resample that sample with replacement Sample multiple times with or without

• ut replacement

Sample once; make strong assumptions about population

Sample Mean and Confidence Intervals for Normally Distributed Data

Estimating Population Statistic

Bootstrap Approach Conventional Approach Sample population

• nce; calculate

sample statistic Sample once; resample that sample with replacement Estimate the mean

Establishing Confidence Intervals

Bootstrap Approach Conventional Approach Sample once; resample that sample with replacement Sample multiple times with or without

• ut replacement

Sample once; make strong assumptions about population Assume population normally distributed

Estimating Population Mean

What is the average height/weight/ income of the population? Common question in science, business, finance Need to estimate mean value of some property of the population Assume population is normally distributed

Normal Distribution

Values close to the mean are more likely than values far away from the mean

Sample A subset - hopefully representative - of the population Population All the data out there in the universe

Draw Sample from Population

Mean and Variance

x1 x2 xn

x

• These statistics only apply to the sample of data,

and so are known as sample statistics The corresponding figures for all possible data points out there are called population statistics

x

• = x1 + x2 + … + xn

n

Variance = xi - x _

( )2

Σ

n-1

From Sample to Population

Population Mean Sample Mean

x

• =

x1 + x2 + … + xn n

μ = ?

Estimating Population Mean

Aim: Estimate a statistical property (mean) of the population Will need to do so from a sample Use properties of sample to estimate property of population

Sampling Distribution

Tricky part is going from properties of sample to property of population Can’t be completely sure of population property Can however be sure of probability distribution of the population property This distribution depends on sample alone - Sampling Distribution

Sampling Distribution

Probability distribution of a population statistic (e.g. population mean), given a particular sample.

From Sample to Population

Population Mean Sample Mean

x

• =

x1 + x2 + … + xn n

μ = ?

From Sample to Population

Population Mean Sample Mean

x

• =

x1 + x2 + … + xn n

Sampling Distribution

Population Mean Sample Mean

x

• =

x1 + x2 + … + xn n

Estimating Population Mean

Turns out, x is the best estimate of μ (Law of Large Numbers) Sample mean is best, unbiased estimator of the population mean Even so, how sure are we of our estimate? Confidence levels help answer this question

“We can be 99% confident that the average is between ___ and ____” Confidence Intervals

Sampling Distribution

Population mean μ has a distribution called the sampling distribution This is a normal distribution

• Mean = Sample mean
• Variance ≈ Sample variance / n
• Std dev. = Sample std dev. / sqrt(n)

68% Confidence That μ is Within 1σ of x

68% _ X _

68% Confidence That μ is Within 1σ of x

68% X _

• 1.s/

n X _

+ 1.s/

n X _ _

68% Confidence That μ is Within 1σ of x

68% X _

• 1.s/

n X _

+ 1.s/

n X _

We can state with 68% confidence that the population mean μ lies in the range

• 1.s/

n X _

+ 1.s/

n X _

to

_

99% X _

99% Confidence That μ is Within 2.57σ of x

_

99% X _

• 2.576s/ n

X _

+ 2.576s/ n

X _

99% Confidence That μ is Within 2.57σ of x

_

99% X _

• 2.576s/ n

X _

+ 2.576s/ n

X _

We can state with 99% confidence that the population mean μ lies in the range

• 2.576s/

n X _

to + 2.576s/

n X _

99% Confidence That μ is Within 2.57σ of x

_

99% X _

• Z. s/

n X _

+ Z. s/

n X _

(100-p)% Confidence That μ is Within Zσ of x

_

99% X _

• Z. s/

n X _

+ Z. s/

n X _

We can state with (100- p)% confidence that the population mean μ lies in the range

• Z.s/

n X _

to + Z.s/

n X _

(100-p)% Confidence That μ is Within Zσ of x

_

Sampling Distribution

p is the level of significance Z is the number of standard deviations from the mean corresponding to p s and x are calculated from the sample properties

_

Sampling Distribution

Confidence Interval Z

80% 1.282 85% 1.440 90% 1.645 95% 1.960 99% 2.576 99.5% 2.807 99.9% 3.291

Sampling Distribution

Range is centered around sample mean Extends symmetrically on both sides Greater the range, the greater our confidence that estimate lies within it

Sample Mean and Confidence Intervals for Any Data

Estimating Population Statistic

Bootstrap Approach Conventional Approach Sample population

• nce; calculate

sample statistic Sample once; resample that sample with replacement Estimate the mean

Establishing Confidence Intervals

Bootstrap Approach Conventional Approach Sample once; resample that sample with replacement Sample multiple times with or without

• ut replacement

Sample once; make strong assumptions about population Make no assumptions of population distribution but draw a large number of samples from population

Sampling Distribution of the Mean

Tricky part is going from properties of samples to property of population Can’t be completely sure of population property Need to know the probability distribution of the population property Using the Sampling Distribution i.e. distribution of estimates from the samples

Population Sample 1 Sample 2 Sample 3 Sample 4 Sample ∞

Draw many samples, calculate mean of each, plot histogram of these means

Sampling Distribution of the Mean

Population Sample 1 Sample 2 Sample 3 Sample 4 Sample ∞

Confidence Intervals from Non-normal Data

Using the Sampling Distribution of the mean, can calculate confidence intervals of our estimate

Population

Confidence Intervals from Non-normal Data

Population

Confidence Intervals from Non-normal Data

Sample values

. . .

Population

Calculate mean of each sample Repeat multiple times

↓

Confidence Intervals from Non-normal Data

↓

Sample values

. . .

Population

Calculate mean of each sample Repeat multiple times

↓

Confidence Intervals from Non-normal Data

↓

Sample Means

Sample values

. . .

Population

Calculate mean of each sample Repeat multiple times

↓

Confidence Intervals from Non-normal Data

↓

97.5% percentile 2.5% percentile

Sample Means

Sample values

. . .

Establishing Confidence Intervals

Bootstrap Approach Conventional Approach Sample once; resample that sample with replacement Sample multiple times with or without

• ut replacement

Sample once; make strong assumptions about population The Central Limit Theorem can be used to estimate the mean of even non-normally distributed data

Central Limit Theorem

A group of means of N samples drawn from any distribution (even a non-normal distribution) approaches normality as N approaches infinity.

Central Limit Theorem

A group of means of N samples drawn from any distribution (even a non-normal distribution) approaches normality as N approaches infinity.

Central Limit Theorem

A group of means of N samples drawn from any distribution (even a non-normal distribution) approaches normality as N approaches infinity.

Central Limit Theorem

A group of means of N samples drawn from any distribution (even a non-normal distribution) approaches normality as N approaches infinity.

Mean of non-normal population can be estimated easily by sampling Draw N samples, compute mean of each sample Compute mean of these means As N -> ∞ this mean of means approaches population mean

Implication of the Central Limit Theorem

Establishing Confidence Intervals

Bootstrap Approach Conventional Approach Sample once; resample that sample with replacement Sample multiple times with or without

• ut replacement

Sample once; make strong assumptions about population The Central Limit Theorem only applies to a group of means, so computing multiple samples is key

Establishing Confidence Intervals

Bootstrap Approach Conventional Approach Sample once; resample that sample with replacement Sample multiple times with or without

• ut replacement

Sample once; make strong assumptions about population Not a very realistic approach in the real world

Establishing Confidence Intervals

Bootstrap Approach Conventional Approach Sample once; resample that sample with replacement Sample multiple times with or without

• ut replacement

Sample once; make strong assumptions about population

Establishing Confidence Intervals

Bootstrap Approach Conventional Approach Sample once; resample that sample with replacement Sample multiple times with or without

• ut replacement

Sample once; make strong assumptions about population Instead modelers choose only to work with data whose distributions are known

Establishing Confidence Intervals

Bootstrap Approach Conventional Approach Sample once; resample that sample with replacement Sample multiple times with or without

• ut replacement

Sample once; make strong assumptions about population For normally distributed data we can often work with just one sample to estimate mean

Population

Confidence Intervals from Normal Data

Simple random sample

. . .

Population

Calculate mean of sample

Compute confidence intervals analytically ↓

Confidence Intervals from Normal Data

↓

Simple random sample

. . .

Demo

The central limit theorem

Demo

Observing the central limit theorem on a real dataset

Drawbacks of Conventional Methods

Drawbacks of Conventional Methods

Make strong assumptions of the distribution of data Use analytical formulae to estimate statistics based on data distributions The analytical formula may not exist for certain combinations

Drawbacks of Conventional Methods

Need to draw a large number of samples from the population Estimate statistics based on sampling distribution May not be practical or realistic

Estimating Population Statistic

Bootstrap Approach Conventional Approach Sample population

• nce; calculate

sample statistic Sample once; resample that sample with replacement

Establishing Confidence Intervals

Bootstrap Approach Conventional Approach Sample once; resample that sample with replacement Sample multiple times with or without

• ut replacement

Sample once; make strong assumptions about population

Establishing Confidence Intervals

Bootstrap Approach Conventional Approach Sample once; resample that sample with replacement Sample multiple times with or without

• ut replacement

Sample once; make strong assumptions about population

Parametric Method

Establishing Confidence Intervals

Bootstrap Approach Conventional Approach Sample once; resample that sample with replacement Sample multiple times with or without

• ut replacement

Sample once; make strong assumptions about population

Non-parametric Methods

The basic Bootstrap method is non- parametric, however parametric variants exist too

The Bootstrap Method

Population Sample 1 Sample 2 Sample 3 Sample 4 Sample ∞

Conventional Methods

Population

Calculate mean of each sample Repeat multiple times

↓

Confidence Intervals from Non-normal Data

↓

97.5% percentile 2.5% percentile

Sample Means

Sample values

. . .

Population Sample 1 Sample 2 Sample 3 Sample 4 Sample ∞

Bootstrap Method

Draw just one sample from the population

Population Sample 1

Bootstrap Method

Draw just one sample from the population

Population Bootstrap Sample

The Bootstrap Sample

Treat that one sample as if it were the population

Sample 1 Sample 2 Sample 3 Sample 4 Sample ∞

Bootstrap Method

Draw multiple samples from the one sample with replacement

Sample 1 Sample 2 Sample 3 Sample 4 Sample ∞

Bootstrap Method

Each of these samples is sometimes called a Bootstrap Replication

Sample 1 Sample 2 Sample 3 Sample 4 Sample ∞

Estimate Statistics using the Bootstrap Method

With each bootstrap replication calculate the statistic e.g. mean

Sample 1 Sample 2 Sample 3 Sample 4 Sample ∞

Estimate Statistics using the Bootstrap Method

Each estimate from a bootstrapped replication is called a bootstrap realization of the statistic

Sample 1 Sample 2 Sample 3 Sample 4 Sample ∞

Confidence Intervals using the Bootstrap Method

Calculate confidence intervals using the bootstrap distribution of the statistic

Sampling with replacement is essential Else each Bootstrap Replication will merely reproduce the Bootstrap Sample

Sampling with Replacement

Reusing the same data multiple times “Bootstrapping” comes from the phrase “pulling yourself up by your

• wn bootstraps”

Has empirically been shown to produce meaningful results

Bootstrapping does not create new data Creates the samples that could have been drawn from the original population Assumes that the bootstrap sample accurately represents the population

Sampling with Replacement

The Bootstrap Method seems like cheating, but it is both theoretically sound and very robust

The Bootstrap Method and Confidence Intervals

Confidence Intervals with the Bootstrap Method

Bootstrap Sample (treated as Population)

Sample values with replacement

. . .

Confidence Intervals with the Bootstrap Method

Bootstrap Sample (treated as Population)

Calculate mean of each sample Repeat multiple times

↓

Confidence Intervals with the Bootstrap Method

↓

Bootstrap Sample (treated as Population)

Sample values with replacement

. . .

Confidence Intervals with the Bootstrap Method

Sample Means Bootstrap Sample (treated as Population)

Calculate mean of each sample Repeat multiple times

↓ ↓

Sample values with replacement

. . .

Confidence Intervals with the Bootstrap Method

Sample Means Bootstrap Sample (treated as Population)

Calculate mean of each sample Repeat multiple times

↓ ↓

Sample values with replacement

. . .

97.5% percentile 2.5% percentile

The Bootstrap Method

Conventional Approach Bootstrap Method Sample population just once if no confidence intervals needed No need to re-sample for confidence intervals for common use-cases Re-sample population if confidence intervals needed for complex cases Sample population just once under all circumstances Re-sample bootstrap sample with replacement under all circumstances No change in procedure, works equally well for common and complex cases

Great for

• Arbitrary population (unknown

distribution)

• Arbitrary statistics (not commonly

studied for arbitrary population)

• Confidence interval around arbitrary

statistics

The Bootstrap Method

Tends to systematically under-estimate variances Various measures to mitigate this bias

• Compute correction based on difference

between bootstrap and sample estimate

• Add back to each bootstrap value
• “Balanced Bootstrap”

Performs poorly for highly skewed data

The Bootstrap Method

Can be used to compute just about any statistic From just about any data However, most widely used to calculate

• Confidence intervals
• Standard errors
• Of complex, hard-to-estimate statistics

The Bootstrap Method

Main use-case of the Bootstrap Method: Calculate confidence interval around a complex statistic

Computing confidence intervals around the mean of a normal distribution

• No need of bootstrap, parametric

method is simpler Computing confidence intervals around the R-squared of a regression

• Bootstrap method is simple, robust, and

effective

The Bootstrap Method

Types of Bootstrap Confidence Intervals

Basic bootstrap Percentile bootstrap Studentized bootstrap Bias-corrected bootstrap Accelerated bootstrap

Summary

Estimating statistics and calculating confidence intervals The Central Limit Theorem Conventional methods vs. bootstrap methods Advantages of bootstrapping techniques

Up Next: Implementing Bootstrap Methods for Summary Statistics