[PPT] - Gov 2000: 8. Simple Linear Regression Matthew Blackwell Fall 2016 PowerPoint Presentation

SLIDE 1

Gov 2000: 8. Simple Linear Regression

Matthew Blackwell

Fall 2016

1 / 84

SLIDE 2

1. Assumptions of the Linear Regression Model
2. Sampling Distribution of the OLS Estimator
3. Sampling Variance of the OLS Estimator
4. Large Sample Properties of OLS
5. Exact Inference for OLS
6. Hypothesis Tests and Confjdence Intervals
7. Goodness of Fit

2 / 84

SLIDE 3

Where are we? Where are we going?

Last week:

▶ Using the CEF to explore relationships ▶ Practical estimation concerns led us to OLS/lines of best fjt.

This week:

▶ Inference for OLS: sampling distribution. ▶ Is there really a relationship? Hypothesis tests ▶ Can we get a range of plausible slope values? Confjdence

intervals

▶ ⇝ how to read regression output. 3 / 84

SLIDE 4

More narrow goal

## ## Call: ## lm(formula = logpgp95 ~ logem4, data = ajr) ## ## Residuals: ## Min 1Q Median 3Q Max ## -2.7130 -0.5333 0.0195 0.4719 1.4467 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 10.6602 0.3053 34.92 < 2e-16 *** ## logem4

0.5641

0.0639

8.83

2.1e-13 *** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 0.756 on 79 degrees of freedom ## (82 observations deleted due to missingness) ## Multiple R-squared: 0.497, Adjusted R-squared: 0.49 ## F-statistic: 78 on 1 and 79 DF, p-value: 2.09e-13

4 / 84

SLIDE 5

1/ Assumptions of the Linear Regression Model

5 / 84

SLIDE 6

Simple linear regression model

We are going to assume a linear model:

𝑍𝑗 = 𝛾0 + 𝛾1𝑌𝑗 + 𝑣𝑗

Data:

▶ Dependent variable: 𝑍𝑗 ▶ Independent variable: 𝑌𝑗

Population parameters:

▶ Population intercept: 𝛾0 ▶ Population slope: 𝛾1

Error/disturbance: 𝑣𝑗

▶ Represents all unobserved error factors infmuencing 𝑍𝑗 other

than 𝑌𝑗.

6 / 84

SLIDE 7

Causality and regression

𝑍𝑗 = 𝛾0 + 𝛾1𝑌𝑗 + 𝑣𝑗

Last week we showed there is always a population linear

regression we called the linear projection.

▶ No notion of causality and may not even be the CEF.

Traditional regression approach: assume slope parameters are

causal or structural.

▶ 𝛾1 is the efgect of a one-unit change in 𝑦 holding all other

factors (𝑣𝑗) constant.

Regression will always consistently estimate a linear

association between 𝑍𝑗 and 𝑌𝑗.

Today: When will regression say something causal?

▶ GOV 2001/2002 has more on a formal language of causality. 7 / 84

SLIDE 8

Linear regression model

In order to investigate the statistical properties of OLS, we

need to make some statistical assumptions:

Linear Regression Model

The observations, (𝑍𝑗, 𝑌𝑗) come from a random (i.i.d.) sample and satisfy the linear regression equation, 𝑍𝑗 = 𝛾0 + 𝛾1𝑌𝑗 + 𝑣𝑗 𝔽[𝑣𝑗|𝑌𝑗] = 0. The independent variable is assumed to have non-zero variance, 𝕎[𝑌𝑗] > 0.

8 / 84

SLIDE 9

Linearity

Assumption 1: Linearity

The population regression function is linear in the parameters: 𝑍𝑗 = 𝛾0 + 𝛾1𝑌𝑗 + 𝑣𝑗

Violation of the linearity assumption:

𝑍𝑗 = 1 𝛾0 + 𝛾1𝑌𝑗 + 𝑣𝑗

Not a violation of the linearity assumption:

𝑍𝑗 = 𝛾0 + 𝛾1𝑌2

𝑗 + 𝑣𝑗

In future weeks, we’ll talk about how to allow for

non-linearities in 𝑌𝑗.

9 / 84

SLIDE 10

Random sample

Assumption 2: Random Sample

We have a iid random sample of size 𝑜, {(𝑍𝑗, 𝑌𝑗) ∶ 𝑗 = 1, 2, … , 𝑜} from the population regression model above.

Violations: time-series, selected samples.
Think about the weight example from last week, where 𝑍𝑗 was

my weight on a given day and 𝑌𝑗 was my number of active minutes the day before: weight𝑗 = 𝛾0 + 𝛾1activity𝑗 + 𝑣𝑗

What if I only weighed myself on the weekdays?

10 / 84

SLIDE 11

A non-iid sample

20 40 60 80 100

3
2
1

X Y

11 / 84

SLIDE 12

Variation in X

Assumption 3: Variation in 𝑌

There is in-sample variation in 𝑌𝑗, so that,

𝑜

∑

𝑗=1

(𝑌𝑗 − 𝑌)2 > 0.

OLS not well-defjned if no in-sample variation in 𝑌𝑗
Remember the formula for the OLS slope estimator:

̂ 𝛾1 = ∑𝑜

𝑗=1(𝑌𝑗 − 𝑌)(𝑍𝑗 − 𝑍)

∑𝑜

𝑗=1(𝑌𝑗 − 𝑌)2

What happens here when 𝑌𝑗 doesn’t vary?

12 / 84

SLIDE 13

Stuck in a moment

Why does this matter? How would you draw the line of best

fjt through this scatterplot, which is a violation of this assumption?

3
2
1

1 2 3

2
1

1 2 X Y

13 / 84

SLIDE 14

Stuck in a moment

Why does this matter? How would you draw the line of best

fjt through this scatterplot, which is a violation of this assumption?

3
2
1

1 2 3

2
1

1 2 X Y

14 / 84

SLIDE 15

Zero conditional mean

Assumption 4: Zero conditional mean of the errors

The error, 𝑣𝑗, has expected value of 0 given any value of the independent variable: 𝔽[𝑣𝑗|𝑌𝑗 = 𝑦] = 0 ∀𝑦.

⇝ weaker condition that 𝑣𝑗 and 𝑌𝑗 uncorrelated:

Cov[𝑣𝑗, 𝑌𝑗] = 𝔽[𝑣𝑗𝑌𝑗] = 0

⇝ 𝔽[𝑍𝑗|𝑌𝑗] = 𝛾0 + 𝛾1𝑌𝑗 is the CEF

15 / 84

SLIDE 16

Violating the zero conditional mean assumption

How does this assumption get violated? Let’s generate data

from the following model: 𝑍𝑗 = 1 + 0.5𝑌𝑗 + 𝑣𝑗

But let’s compare two situations:
1. Where the mean of 𝑣𝑗 depends on 𝑌𝑗 (they are correlated)
2. No relationship between them (satisfjes the assumption)
3
2
1

1 2 3

2
1

1 2 3 4 5

Assumption 4 violated

X Y

3
2
1

1 2 3

2
1

1 2 3 4 5

Assumption 4 not violated

X Y 16 / 84

SLIDE 17

More examples of zero conditional mean in the error

Think about the weight example from last week, where 𝑍𝑗 was

my weight on a given day and 𝑌𝑗 was my number of active minutes the day before: weight𝑗 = 𝛾0 + 𝛾1activity𝑗 + 𝑣𝑗

What might in 𝑣𝑗 here? Amount of food eaten, workload, etc

etc.

We have to assume that all of these factors have the same

mean, no matter what my level of activity was. Plausible?

When is this assumption most plausible? When 𝑌𝑗 is randomly

assigned.

17 / 84

SLIDE 18

2/ Sampling Distribution of the OLS Estimator

18 / 84

SLIDE 19

What is OLS?

Ordinary least squares (OLS) is an estimator for the slope and

the intercept of the regression line.

Where does it come from? Minimizing the sum of the squared

residuals: ( ̂ 𝛾0, ̂ 𝛾1) = arg min

𝑐0,𝑐1 𝑜

∑

𝑗=1

(𝑍𝑗 − 𝑐0 − 𝑐1𝑌𝑗)2

Leads to:

̂ 𝛾0 = 𝑍 − ̂ 𝛾1𝑌 ̂ 𝛾1 = ∑𝑜

𝑗=1(𝑌𝑗 − 𝑌)(𝑍𝑗 − 𝑍)

∑𝑜

𝑗=1(𝑌𝑗 − 𝑌)2

19 / 84

SLIDE 20

Intuition of the OLS estimator

Regression line goes through the sample means (𝑍, 𝑌):

𝑍 = ̂ 𝛾0 + ̂ 𝛾1𝑌

Slope is the ratio of the covariance to the variance of 𝑌𝑗:

̂ 𝛾1 = ∑𝑜

𝑗=1(𝑌𝑗 − 𝑌)(𝑍𝑗 − 𝑍)

∑𝑜

𝑗=1(𝑌𝑗 − 𝑌)2

= ̂ Cov(𝑌𝑗, 𝑍𝑗) ̂ 𝕎[𝑌𝑗] = Sample Covariance between 𝑌 and 𝑍 Sample Variance of 𝑌

20 / 84

SLIDE 21

The sample linear regression function

The estimated or sample regression function is:

̂ 𝑍𝑗 = ̂ 𝛾0 + ̂ 𝛾1𝑌𝑗

Estimated intercept: ̂

𝛾0

Estimated slope: ̂

𝛾1

Predicted/fjtted values: ̂

𝑍𝑗

Residuals:

̂ 𝑣𝑗 = 𝑍𝑗 − ̂ 𝑍𝑗

You can think of the residuals as the prediction errors of our

estimates.

21 / 84

SLIDE 22

OLS slope as a weighted sum of the outcomes

One useful derivation that we’ll do moving forward is to write

the OLS estimator for the slope as a weighted sum of the

utcomes.

̂ 𝛾1 =

𝑜

∑

𝑗=1

𝑋𝑗𝑍𝑗

Where here we have the weights, 𝑋𝑗 as:

𝑋𝑗 = (𝑌𝑗 − 𝑌) ∑𝑜

𝑗=1(𝑌𝑗 − 𝑌)2

Estimation error:

proof

̂ 𝛾1 − 𝛾1 =

𝑜

∑

𝑗=1

𝑋𝑗𝑣𝑗

⇝ ̂

𝛾1 is a sum of random variables.

22 / 84

SLIDE 23

Sampling distribution of the OLS estimator

Remember: OLS is an estimator—it’s a machine that we plug

data into and we get out estimates. OLS

Sample 1: {(𝑍1, 𝑌1), … , (𝑍𝑜, 𝑌𝑜)} (̂ 𝛾0, ̂ 𝛾1)1 Sample 2: {(𝑍1, 𝑌1), … , (𝑍𝑜, 𝑌𝑜)} (̂ 𝛾0, ̂ 𝛾1)2

⋮ ⋮

Sample 𝑙 − 1: {(𝑍1, 𝑌1), … , (𝑍𝑜, 𝑌𝑜)} (̂ 𝛾0, ̂ 𝛾1)𝑙−1 Sample 𝑙: {(𝑍1, 𝑌1), … , (𝑍𝑜, 𝑌𝑜)} (̂ 𝛾0, ̂ 𝛾1)𝑙

Just like the sample mean, sample difgerence in means, or the

sample variance

It has a sampling distribution, with a sampling

variance/standard error, etc.

23 / 84

SLIDE 24

Simulation procedure

Let’s take a simulation approach to demonstrate:

▶ Pretend that the AJR data represents the population of

interest

▶ See how the line varies from sample to sample

1. Draw a random sample of size 𝑜 = 30 with replacement using

sample()

2. Use lm() to calculate the OLS estimates of the slope and

intercept

3. Plot the estimated regression line

24 / 84

SLIDE 25