Gov 2000: 7. What is Regression? Matthew Blackwell Fall 2016 1 / - - PowerPoint PPT Presentation

Gov 2000: 7. What is Regression?

Matthew Blackwell

Fall 2016

1 / 65

• 1. Relationships between Variables
• 2. Conditional Expectation
• 3. Estimating the CEF
• 4. Linear CEFs and Linear Projections
• 5. Least Squares

2 / 65

Where are we? Where are we going?

• What we’ve been up to: estimating parameters of population
• distributions. Generally we’ve been learning about a single

variable.

• This week and for the rest of the term, we’ll be interested in

the relationships between variables. How does one variable change we change the values of another variable? These will be the bread and butter of the class moving forward.

3 / 65

AJR data

2 4 6 8 10 6 7 8 9 10 Average Protection Against Expropriation Risk Log GDP pc, 1995

• How do we draw this line?

4 / 65

1/ Relationships between Variables

5 / 65

What is a relationship and why do we care?

• Most of what we want to do in the social science is learn

about how two variables are related

• Examples:

▶ Does turnout vary by types of mailers received? ▶ Is the quality of political institutions related to average

incomes?

▶ Does confmict mediation help reduce civil confmict? 6 / 65

Notation and conventions

• 𝑍𝑗 - the dependent variable or outcome or regressand or

left-hand-side variable or response

▶ Voter turnout ▶ Log GDP per capita ▶ Number of battle deaths

• 𝑌𝑗 - the independent variable or explanatory variable or

regressor or right-hand-side variable or treatment or predictor

▶ Social pressure mailer versus Civic Duty Mailer ▶ Average Expropriation Risk ▶ Presence of confmict mediation 7 / 65

Joint distribution review

• (𝑍𝑗, 𝑌𝑗) are draws from an i.i.d. joint distribution 𝑔𝑍,𝑌

▶ 𝑍𝑗 and 𝑌𝑗 are measured on the same unit 𝑗 ▶ WARNING difgerent than our use of 𝑍𝑗 and 𝑌𝑗 as r.v.s for

difgerent groups.

▶ There, 𝑍𝑗 and 𝑌𝑗 corresponded to difgerent units.

• Several ways to summarize the joint population distribution:

▶ Covariance/correlation ▶ Conditional expectation

• Today we’ll spend a lot of time thinking about the relevant

populations parameters for estimating relationships.

▶ Population-fjrst approach. 8 / 65

2/ Conditional Expectation

9 / 65

Conditional expectation function

• Conditional expectation function (CEF): how the mean of 𝑍𝑗

changes as 𝑌𝑗 changes. 𝜈(𝑦) = 𝔽[𝑍𝑗|𝑌𝑗 = 𝑦]

• The CEF is a feature of the joint distribution of 𝑍𝑗 and 𝑌𝑗:

𝔽[𝑍𝑗|𝑌𝑗 = 𝑦] = ∫

∞ −∞ 𝑧𝑔𝑍|𝑌(𝑧|𝑦)𝑒𝑧

• Goal of regression is to estimate CEF: ̂

𝜈(𝑦) = ̂ 𝔽[𝑍𝑗|𝑌𝑗 = 𝑦]

10 / 65

CEF for binary covariates

• Example:

▶ 𝑍𝑗 is the time respondent 𝑗 waited in line to vote. ▶ 𝑌𝑗 = 1 for whites, 𝑌𝑗 = 0 for non-whites.

• Then the mean in each group is just a conditional expectation:

𝜈(white) = 𝐹[𝑍𝑗|𝑌𝑗 = white] 𝜈(non-white) = 𝐹[𝑍𝑗|𝑌𝑗 = non-white]

• Notice here that since 𝑌𝑗 can only take on two values, 0 and 1,

then these two conditional means completely summarize the CEF.

11 / 65

Why is the CEF useful?

10 20 30 40 50 60 Voting Wait Time

Whites Non-whites

μ(1) μ(0)

• The CEF encodes relationships between variables.
• If 𝜈(white) < 𝜈(non-white), so that waiting times for whites

are shorter on average than for non-whites.

• Indicates a relationship in the population between race and

wait times.

12 / 65

CEF for discrete covariates

• New covariate: 𝑌𝑗 is the number of polling booths at citizen

𝑗’s polling station.

• The mean of 𝑍𝑗 changes as 𝑌𝑗 changes:

5 10 15 Voting Wait Time

5 Booths

μ(5)

10 Booths

μ(10)

15 Booths

μ(15)

20 Booths

μ(20) 13 / 65

CEF with multiple covariates

• We could also be interested in the CEF conditioning on

multiple variables: 𝜈(white, man) = 𝔽[𝑍𝑗|𝑌𝑗 = white, 𝑎𝑗 = man] 𝜈(white, woman) = 𝔽[𝑍𝑗|𝑌𝑗 = white, 𝑎𝑗 = woman] 𝜈(non-white, man) = 𝔽[𝑍𝑗|𝑌𝑗 = non-white, 𝑎𝑗 = man] 𝜈(non-white, woman) = 𝔽[𝑍𝑗|𝑌𝑗 = non-white, 𝑎𝑗 = woman]

• Why? Allows more credible all else equal comparisons (ceteris

paribus).

• Ex: average difgerence in wait times between white and

non-white citizens of the same gender: 𝜈(white, man) − 𝜈(non-white, man)

14 / 65

CEF for continuous covariates

• What if our independent variable, 𝑌𝑗 is income?
• Many possible values of 𝑌𝑗 ⇝ many possible values of

𝔽[𝑍𝑗|𝑌𝑗 = 𝑦].

▶ Writing out each value of the CEF no longer feasible.

• Now we will think about 𝜈(𝑦) = 𝔽[𝑍𝑗|𝑌𝑗 = 𝑦] as function.

What does this function look like:

▶ Linear: 𝜈(𝑦) = 𝛽 + 𝛾𝑦 ▶ Quadratic: 𝜈(𝑦) = 𝛽 + 𝛾𝑦 + 𝛿𝑦2 ▶ Crazy, nonlinear: 𝜈(𝑦) = 𝛽/(𝛾 + 𝑦)

• These are unknown functions in the population! This is going

to make producing an estimator ̂ 𝜈(𝑦) very diffjcult!

15 / 65

Wait times and income

Income Wait Times 20 40 60 80 $25k $50k 100k 150k 200k

16 / 65

Wait times and income

Income Wait Times 20 40 60 80 $25k $50k 100k 150k 200k 30 60 Wait Times μ($25k)

17 / 65

Wait times and income

Income Wait Times 20 40 60 80 $25k $50k 100k 150k 200k 30 60 Wait Times μ($25k) μ($50k)

18 / 65

Wait times and income

Income Wait Times 20 40 60 80 $25k $50k 100k 150k 200k 30 60 Wait Times μ($25k) μ($50k) μ($75k)

19 / 65

Wait times and income

Income Wait Times 20 40 60 80 $25k $50k 100k 150k 200k 30 60 Wait Times μ($25k) μ($50k) μ($75k) μ($150k)

20 / 65

The CEF decomposition

• We can always decompose 𝑍𝑗 into the CEF and an error:

𝑍𝑗 = 𝔽[𝑍𝑗|𝑌𝑗] + 𝑣𝑗

• Here, the CEF error has two defjnitional properties:

▶ The mean of the error doesn’t depend on 𝑌𝑗:

𝔽[𝑣𝑗|𝑌𝑗] = 𝔽[𝑣𝑗] = 0

▶ The error is uncorrelated with any function of 𝑌𝑗.

• 𝑍𝑗 can be decomposed into the part “explained by 𝑌𝑗” and a

part that is uncorrelated with 𝑌𝑗.

21 / 65

Best predictor

• Another reason to focus on the CEF: it generates best

predictions about 𝑍𝑗 using 𝑌𝑗.

• Let 𝑕(𝑌𝑗) be some function that generates prediction and

defjne the mean squared error (MSE) of the prediction as: 𝔽[(𝑍𝑗 − 𝑕(𝑌𝑗))2]

• What function should you pick? The CEF minimizes this

prediction error: 𝔽[(𝑍𝑗 − 𝑕(𝑌𝑗))2] ≥ 𝔽[(𝑍𝑗 − 𝜈(𝑌𝑗))2]

• We say the CEF is the best predictor of 𝑍𝑗 among functions of

𝑌𝑗.

▶ …in terms of squared error. 22 / 65

3/ Estimating the CEF

23 / 65

Estimating the CEF for binary covariates

• How do we estimate ̂

𝔽[𝑍𝑗|𝑌𝑗 = 𝑦]?

• Sample means within each group:

̂ 𝔽[𝑍𝑗|𝑌𝑗 = 1] = 1 𝑜1 ∑

𝑗∶𝑌𝑗=1

𝑍𝑗 ̂ 𝔽[𝑍𝑗|𝑌𝑗 = 0] = 1 𝑜0 ∑

𝑗∶𝑌𝑗=0

𝑍𝑗

• 𝑜1 = ∑𝑜

𝑗=1 𝑌𝑗 is the number of women in the sample.

• 𝑜0 = 𝑜 − 𝑜1 is the number of men.
• ∑𝑗∶𝑌𝑗=1 sum only over the 𝑗 that have 𝑌𝑗 = 1, meaning that 𝑗

is a woman.

• ⇝ estimate the mean of 𝑍𝑗 conditional on 𝑌𝑗 by just

estimating the means within each group of 𝑌𝑗.

24 / 65

Binary covariate example

## mean of log GDP among non-African countries mean(ajr$logpgp95[ajr$africa == 0], na.rm = TRUE) ## [1] 8.716 ## mean of log GDP among African countries mean(ajr$logpgp95[ajr$africa == 1], na.rm = TRUE) ## [1] 7.355

25 / 65

Binary covariate CEF plot

plot(ajr$africa, ajr$logpgp95, ylab = "Log GDP per capita", xlab = "Africa", bty = "n") points(x = 0, y = mean(ajr$logpgp95[ajr$africa == 0], na.rm = TRUE), pch = 19, col = "red", cex = 3) points(x = 1, y = mean(ajr$logpgp95[ajr$africa == 1], na.rm = TRUE), pch = 19, col = "red", cex = 3) 0.0 0.2 0.4 0.6 0.8 1.0 6 7 8 9 10

Africa Log GDP per capita

26 / 65

Discrete covariate: estimating the CEF

• What if 𝑌𝑗 isn’t binary, but takes on > 2 discrete values?
• The same logic applies, we can still estimate 𝔽[𝑍𝑗|𝑌𝑗 = 𝑦]

with the sample mean among those who have 𝑌𝑗 = 𝑦: ̂ 𝔽[𝑍𝑗|𝑌𝑗 = 𝑦] = 1 𝑜𝑦 ∑

𝑗∶𝑌𝑗=𝑦

𝑍𝑗

27 / 65

Discrete covariate example

• I’ve been collecting data on my own weight for a while.
• How does my weight (𝑍𝑗) varied by the day of the week (𝑌𝑗)?
• Calculate the mean weight for each day of the week:

weight <- read.csv("../data/weight.csv", stringsAsFactors = FALSE) weight$weekday <- as.numeric(format(as.Date(weight$date, format = "%m/%d/%y%n%H:%M"), "%w")) + 1 weight$date <- as.Date(weight$date, format = "%m/%d/%y%n%H:%M") day.means <- rep(NA, times = 7) names(day.means) <- c("1 - Su", "2 - Mo", "3 - Tu", "4 - We", "5 - Th", "6 - Fr", "7 - Sa") for (i in 1:7) { day.means[i] <- mean(weight$weight[weight$weekday == i]) } day.means ## 1 - Su 2 - Mo 3 - Tu 4 - We 5 - Th 6 - Fr 7 - Sa ## 170.4 170.2 169.6 169.5 169.7 169.8 170.2

28 / 65

Discrete covariate CEF plot

plot(x = weight$weekday, y = weight$weight, xaxt = "n", xlab = "Weekday", ylab = "Average Weight", pch = 19, col = "grey60") lines(x = 1:7, y = day.means, pch = 19, col = "indianred", lwd = 3) points(x = 1:7, y = day.means, pch = 21, col = "white", cex = 3, bg = "indianred") axis(side = 1, at = 1:7, labels = names(day.means)) 166 168 170 172 174 176

Weekday Average Weight 1 - Su 2 - Mo 3 - Tu 4 - We 5 - Th 6 - Fr 7 - Sa

29 / 65

Continuous covariate (I): each unique value gets a mean

• What if 𝑌𝑗 is continuous? Can we calculate a mean for every

value of 𝑌𝑗?

• Not really, because remember the probability that two values

will be the same in a continuous variable is 0.

• Thus, we’ll end up with a very “jumpy” function,

̂ 𝔽[𝑍𝑗|𝑌𝑗 = 𝑦], since 𝑜𝑦 will be at most 1 for any value of 𝑦.

30 / 65

Continuous covariate (I) example

• I also wear an activity tracker and that collects how active I

am during the day

• Let’s look at the relationship between my weight and my

active minutes in the previous day using this approach.

fitbit <- read.csv("../data/fitbit.csv", stringsAsFactors = FALSE) fitbit$date <- as.Date(fitbit$date, format = "%m/%d/%y") ## lag fitbit by one day fitbit$date <- fitbit$date + 1 ## merge fitbit and weight data weight <- merge(weight, fitbit, by = "date")

31 / 65

Continuous covariate (I) CEF plot

plot(weight$active.mins[order(weight$active.mins)], weight$weight[order(weight$active.mins)], type = "l", lwd = 3, pch = 19, col = "indianred",xlab = "Active Minutes Previous Day", ylab = "Weight") points(weight$active.mins, weight$weight, pch = 19, cex = 0.5) 20 40 60 80 100 120 166 168 170 172 174

Active Minutes Previous Day Weight

• Not a useful summary of the relationship between 𝑌𝑗 and 𝑍𝑗.

32 / 65

Continuous covariate (II): stratify and take means

• So, that seems like each value of 𝑌𝑗 won’t work, but maybe

we can take the continuous variable and turn it into a discrete

• variable. We call this stratifjcation.
• Once it’s discrete, we can just calculate the means within

each strata.

• For instance, we could break up the “Active Minutes” variable

into 3 categories: lazy (< 30mins), active (30-60mins), and very active (>60min).

lowactivity.mean <- mean(weight$weight[weight$active.mins < 30]) medactivity.mean <- mean(weight$weight[weight$active.mins >= 30 & weight$active.mins < 60]) hiactivity.mean <- mean(weight$weight[weight$active.mins >= 60])

33 / 65

Continuous covariate (II) stratified CEF

20 40 60 80 100 120 166 168 170 172 174 Active Minutes Previous Day Weight

34 / 65

Continuous covariate (II) stratified CEF

20 40 60 80 100 120 166 168 170 172 174 Active Minutes Previous Day Weight

Lazy Active Very Active

35 / 65

Continuous covariate (II) stratified CEF

20 40 60 80 100 120 166 168 170 172 174 Active Minutes Previous Day Weight

Lazy Active Very Active

36 / 65

4/ Linear CEFs and Linear Projections

37 / 65

Linear CEFs

• Obviously, estimation is going to be diffjcult with continuous

covariates.

▶ Even stratifjcation had many hidden assumptions: number of

categories, cutofgs for the categories, constant means within strata, etc.

• We can side-step some of these issues by assuming that the

CEF is linear: 𝜈(𝑦) = 𝔽[𝑍𝑗|𝑌𝑗 = 𝑦] = 𝛾0 + 𝛾1𝑦

• Intercept, 𝛾0: the condition expectation of 𝑍𝑗 when 𝑌𝑗 = 0
• Slope, 𝛾1: average change in the mean of 𝑍𝑗 given a one-unit

change in 𝑌𝑗

38 / 65

Why is linearity an assumption?

• Example: 𝑍𝑗 is income, 𝑌𝑗 is years of education.

▶ 𝛾0: average income among people with 0 years of education. ▶ 𝛾1: expected difgerence in income between two adults that

difger by 1 year of education.

• Why is linearity an assumption?

𝔽[𝑍𝑗|𝑌𝑗 = 12]−𝔽[𝑍𝑗|𝑌𝑗 = 11] = 𝔽[𝑍𝑗|𝑌𝑗 = 16]−𝔽[𝑍𝑗|𝑌𝑗 = 15] = 𝛾1

• Efgect of getting HS degree is the same as the efgect of

getting college degree.

39 / 65

Linear CEF with a binary covariate

• Return to wait-times and race example, with 𝑌𝑗 = 1 being

white and 𝑌𝑗 = 0 being non-white.

▶ Two possible values of the CEF: 𝜈(1) for whites and 𝜈(0) for

non-whites.

• Can write the CEF as follows:

𝔽[𝑍𝑗|𝑌𝑗 = 𝑦] = 𝜈(𝑦) = 𝜈(0) + (𝜈(1) − 𝜈(0)) 𝑦

• Rewriting with 𝛾0 = 𝜈(0) and 𝛾1 = 𝜈(1) − 𝜈(0):

𝜈(𝑦) = 𝛾0 + 𝛾1𝑦

• No assumptions, just rewriting!

▶ 𝛾0: expected wait-time for non-whites ▶ 𝛾1: difgerence in expected wait times between whites and

non-whites.

40 / 65

Linear approximation

• Ugh, what if the CEF isn’t linear but we assume it is?
• Better to think of there being a population line of best fjt that

is the best linear approximation to 𝑍𝑗.

• Mathematically, fjnd the linear function of 𝑌𝑗 that minimizes

the squared prediction errors: (𝛾0, 𝛾1) = arg min

(𝑐0,𝑐1)

𝔽[(𝑍𝑗 − (𝑐0 + 𝑐1𝑌𝑗))2]

• Resulting function 𝛾0 + 𝛾1𝑌𝑗 is called the linear projection or

the population linear regression of 𝑍𝑗 onto 𝑌𝑗.

• In general, distinct from the CEF:

▶ CEF, 𝜈(𝑦) is the best predictor of 𝑍𝑗 among all functions. ▶ Linear projection is best predictor among linear functions. 41 / 65

Linear approximation

Income Wait Times 20 40 60 80 $25k $50k 100k 150k 200k

42 / 65

Linear approximation

Income Wait Times 20 40 60 80 $25k $50k 100k 150k 200k CEF

43 / 65

Linear approximation

Income Wait Times 20 40 60 80 $25k $50k 100k 150k 200k CEF Linear Projection

44 / 65

Population linear projection

• Can we relate the intercept and slope of the population line of

best fjt to the joint distribution of 𝑍𝑗 and 𝑌𝑗?

• Yes, using some multivariate calculus, can show:

𝛾0 = 𝔽[𝑍𝑗] − 𝛾1𝔽[𝑌𝑗] 𝛾1 = Cov[𝑍𝑗, 𝑌𝑗] 𝕎[𝑌𝑗]

• What’s awesome about the linear projection is that it exists

and is well-defjned even if the CEF is nonlinear.

45 / 65

Why the linear projection?

• Two handy results about the linear projection:

CEF is linear

If the CEF is a linear function, 𝔽[𝑍𝑗|𝑌𝑗] = 𝑐0 + 𝑐1𝑌𝑗, then it will be equal to the linear projection: 𝔽[𝑍𝑗|𝑌𝑗] = 𝛾0 + 𝛾1𝑌𝑗.

Linear projection approximates CEF

The linear projection is the best linear approximation to the CEF, so that: (𝛾0, 𝛾1) = arg min

(𝑐0,𝑐1)

𝔽[(𝜈(𝑌𝑗) − (𝑐0 + 𝑐1𝑌𝑗))2]

46 / 65

5/ Least Squares

47 / 65

Back up and review

• To review our approach:

▶ Defjned a population line of best fjt, 𝛾0 + 𝛾1𝑌𝑗. ▶ If CEF is linear, it is equal to this line.

• Either way, 𝛾0 and 𝛾1 are valid population parameters just

like 𝜈 or 𝜏2!

• Sample: {(𝑍1, 𝑌1), … , (𝑍𝑜, 𝑌𝑜)} are i.i.d. draws from a

population joint distribution, 𝑔(𝑍,𝑌)(𝑧, 𝑦)

• How can we use this sample to estimate 𝛾0, 𝛾1?

48 / 65

Sample line of best fit

• To get the linear projection, we found the population line of

best fjt: (𝛾0, 𝛾1) = arg min

(𝑐0,𝑐1)

𝔽[(𝑍𝑗 − 𝑐0 − 𝑐1𝑌𝑗)2]

• To get the sample line of best fjt, we replace the population

expectation with a sample mean: ( ̂ 𝛾0, ̂ 𝛾1) = arg min

𝑐0,𝑐1

1 𝑜

𝑜

∑

𝑗=1

(𝑍𝑗 − 𝑐0 − 𝑐1𝑌𝑗)2

• This estimator is called least squares (LS) or ordinary least

squares (OLS).

49 / 65

Fitued OLS lines

2 4 6 8 10 6 7 8 9 10 Average Protection Against Expropriation Risk Log GDP per capita

β0 + β1Xi (OLS Line)

50 / 65

Fitued OLS line

2 4 6 8 10 6 7 8 9 10 Average Protection Against Expropriation Risk Log GDP per capita

β0 + β1Xi (OLS Line) β0 + β1Xi (Alt. Line)

51 / 65

Fitued values and residuals

• Defjnition A fjtted value is the estimated conditional mean of

𝑍𝑗 for a particular observation with independent variable 𝑌𝑗: ̂ 𝑍𝑗 = ̂ 𝔽[𝑍𝑗|𝑌𝑗] = ̂ 𝛾0 + ̂ 𝛾1𝑌𝑗

• Defjnition The residual is the difgerence between the actual

value of 𝑍𝑗 and the fjtted value, ̂ 𝑍𝑗: ̂ 𝑣𝑗 = 𝑍𝑗 − ̂ 𝑍𝑗 = 𝑍𝑗 − ̂ 𝛾0 − ̂ 𝛾1𝑌𝑗

52 / 65

Fitued OLS line

2 4 6 8 10 6 7 8 9 10 Average Protection Against Expropriation Risk Log GDP per capita

β0 + β1Xi IRN

Yi

53 / 65

Fitued OLS line

2 4 6 8 10 6 7 8 9 10 Average Protection Against Expropriation Risk Log GDP per capita

IRN β0 + β1Xi

Yi Yi

54 / 65

Fitued OLS line

2 4 6 8 10 6 7 8 9 10 Average Protection Against Expropriation Risk Log GDP per capita

IRN β0 + β1Xi

Yi Yi ui = Yi − Yi

55 / 65

Why not this line?

2 4 6 8 10 6 7 8 9 10 Average Protection Against Expropriation Risk Log GDP per capita

β0 + β1Xi (Alt. Line) IRN

ui = Yi − Yi ≈ 0

56 / 65

Minimize the residuals

• The residuals,

̂ 𝑣𝑗 = 𝑍𝑗 − ̂ 𝛾0 − ̂ 𝛾1𝑌𝑗, tell us how well the line fjts the data.

▶ Larger magnitude residuals means that points are very far from

the line

▶ Residuals close to 0 mean points very close to the line

• The smaller the magnitude of the residuals, the better we are

doing at predicting 𝑍𝑗

• Choose the line that minimizes the residuals

57 / 65

Which is betuer at minimizing residuals?

2 4 6 8 10 6 7 8 9 10 Average Protection Against Expropriation Risk Log GDP per capita

β0 + β1Xi (OLS Line) β0 + β1Xi (Alt. Line)

58 / 65

OLS estimator

• OLS estimator defjned by minimized squared residuals:

( ̂ 𝛾0, ̂ 𝛾1) = arg min

𝑐0,𝑐1

1 𝑜

𝑜

∑

𝑗=1

(𝑍𝑗 − 𝑐0 − 𝑐1𝑌𝑗)2

• Can we write the OLS intercept ( ̂

𝛾0) and slope ( ̂ 𝛾1) in terms

• f quantities that we know?

Yes! ̂ 𝛾0 = 𝑍 − ̂ 𝛾1𝑌 ̂ 𝛾1 = ∑𝑜

𝑗=1(𝑍𝑗 − 𝑍)(𝑌𝑗 − 𝑌)

∑𝑜

𝑗=1(𝑌𝑗 − 𝑌)2

59 / 65

Sample (co)variance

• Sample covariance:

̂ Cov[𝑌, 𝑍] = 1 𝑜 − 1

𝑜

∑

𝑗=1

(𝑌𝑗 − 𝑌𝑜)(𝑍𝑗 − 𝑍𝑜)

• Sample variance:

̂ 𝕎[𝑌𝑗] = 1 𝑜 − 1

𝑜

∑

𝑗=1

(𝑌𝑗 − 𝑌)2

• Thus, we can rewrite the OLS slope as:

̂ 𝛾1 = ∑𝑜

𝑗=1(𝑍𝑗 − 𝑍)(𝑌𝑗 − 𝑌)

∑𝑜

𝑗=1(𝑌𝑗 − 𝑌)2

= ̂ Cov(𝑌, 𝑍) ̂ 𝕎[𝑌𝑗]

60 / 65

Linear projection vs OLS

• Compare the linear projection intercept/slope in the

population: 𝛾0 = 𝔽[𝑍𝑗] − 𝛾1𝔽[𝑌𝑗] 𝛾1 = Cov[𝑍𝑗, 𝑌𝑗] 𝕎[𝑌𝑗]

• With the OLS intercept/slope in the sample:

̂ 𝛾0 = 𝑍 − ̂ 𝛾1𝑌 ̂ 𝛾1 = ̂ Cov(𝑌, 𝑍) ̂ 𝕎[𝑌𝑗]

• OLS is just replaces all the population expectations with

sample versions!

61 / 65

AJR Example in R

• Let’s use those simple formulas we just learned:

ajr <- na.omit(ajr[, c("avexpr", "logpgp95")]) cov.xy <- cov(ajr$avexpr, ajr$logpgp95) var.x <- var(ajr$avexpr) cov.xy/var.x ## [1] 0.5319 mean(ajr$logpgp95) - cov.xy/var.x * mean(ajr$avexpr) ## [1] 4.626

• Compare it to what lm(), the OLS function in R produces:

coef(lm(logpgp95 ~ avexpr, data = ajr)) ## (Intercept) avexpr ## 4.6261 0.5319

62 / 65

Mechanical properties of least squares

• The residuals will be 0 on average:

𝑜

∑

𝑗=1

̂ 𝑣𝑗 = 0

• The residuals will be uncorrelated with the predictor:

𝑜

∑

𝑗=1

𝑌𝑗 ̂ 𝑣𝑗 = 0 ⇝ ̂ Cov(𝑌𝑗, ̂ 𝑣𝑗) = 0

• The residuals will be uncorrelated with the fjtted values:

𝑜

∑

𝑗=1

̂ 𝑍𝑗 ̂ 𝑣𝑗 = 0 ⇝ ̂ Cov(̂ 𝑍𝑗, ̂ 𝑣𝑗) = 0

63 / 65

Mechanical properties of least squares in R

mod <- lm(logpgp95 ~ avexpr, data = ajr) mean(residuals(mod)) ## [1] -2.006e-17 cor(ajr$logem4, residuals(mod)) ## [1] -3.185e-17 cor(fitted(mod), residuals(mod)) ## [1] -1.16e-16

64 / 65

Wrap up

• What is regression: estimating the CEF of 𝑍𝑗 given 𝑌𝑗
• Easy to do with sample means with discrete 𝑌𝑗
• Need parametric assumptions when 𝑌𝑗 is continuous
• Derived an estimator for linear projection of 𝑍𝑗 on 𝑌𝑗

65 / 65