Applied Statistics Lecturer: Serena Arima Regression model Model - - PowerPoint PPT Presentation

Regression model Model estimation Properties OLS estimator

Applied Statistics

Lecturer: Serena Arima

Regression model Model estimation Properties OLS estimator

Linear regression model

Consider the income and the expenditure of a sample of n = 1000 italian individuals in 2010. Possible questions:

1 Are the income and the expenditure related? 2 How are these variables related? 3 Can we use these variables to predict the expenditure of

the next year corresponding to a fixed income?

Regression model Model estimation Properties OLS estimator

Linear regression model

A linear regression model can be specified as follows yi = β0 + β1xi1 + β2xi2 + ... + βkxik + ǫi = x′

i β + ǫi

where yi is the response variable; xi1, .., xik are the explanatory variables or predictors; ǫi is the unobserved random term.

Regression model Model estimation Properties OLS estimator

Linear regression model

The equality yi = x′

i β + ǫi

is supposed to hold for any possible observation, while we only

• bserve a sample of n observations.

We shall consider this sample as one realization of all potential samples of size n that have been drawn from the same population.

In this way, we can view yi and ǫi as random variables. In the regression context, we consider the predictors as observed and fixed (deterministic). Using the matrix notation, Y = Xβ + ǫ

Regression model Model estimation Properties OLS estimator

Linear regression model

Mathematical model ↔ Statistical model

• 20

40 60 80 100 2000 4000 6000 8000 10000 Change of measurement scale y=100*x Length in m Length in cm

• ●
• ●
• ●
• 20

40 60 80 100 50 100 150 200 Statistical model Height and weight measurements Weight Height

Regression model Model estimation Properties OLS estimator

Linear regression model

Our goal is to find the coefficient of the linear combination ˜ β0, ˜ β1, ˜ β2... ˜ βk that minimize the following objective function

S(˜ β) =

N

• i=1

(yi − ˜ β0 − ˜ β1x1i − ˜ β2x2i − ... − ˜ βkxki)2 =

N

• i=1

(yi − x′

i ˜

β)2 That is we minimize the sum of squared approximation errors. This approach is referred to ordinary least squares or OLS approach.

Regression model Model estimation Properties OLS estimator

Simple linear model

Suppose we want to estimate a simple regression model yi = β0 + β1xi + ǫi that is we want to estimate the regression line

• yi =

β0 + β1xi such that S( β0, β1) =

n

• i=1

(yi − yi)2 =

n

• i=1

e2

i

is minimum.

1

• yi predicted values;

2 ei residuals.

Regression model Model estimation Properties OLS estimator

Simple linear model

Suppose we want to estimate a simple regression model yi = β0 + β1xi + ǫi that is we want to estimate the regression line

• yi =

β0 + β1xi such that S( β0, β1) =

n

• i=1

(yi − yi)2 =

n

• i=1

e2

i

is minimum.

1

• yi predicted values;

2 ei residuals.

Regression model Model estimation Properties OLS estimator

Simple linear model

We can estimate β0 and β1 with the OLS method. dS( β0, β1) dβ0 = −2

n

• i=1

(yi − β0 − β1xi) = 0 dS( β0, β1) dβ1 = −2

n

• i=1

xi(yi − β0 − β1xi) = 0

Regression model Model estimation Properties OLS estimator

Simple linear model

Solving the system we get:

• β0

= ¯ y − β1¯ x

• β1

= n

i=1(xi − ¯

x)(yi − ¯ y) n

i=1(xi − ¯

x) = n

i=1 xiyi − n¯

x ¯ y n

i=1 x2 i − n¯

x2 = Cov(X, Y )

• Var(x)

= ρ

• Var(Y )

Var(X)

Regression model Model estimation Properties OLS estimator

Example 1

Italian income and expenditure data1: we have selected a sample of 1000 of italians and we have collected the following variables: Income (annual income); Expenditure (annual expenditure); Age; Number of components in the family.

1Data from Banca d’Italia

Regression model Model estimation Properties OLS estimator

Example 1

We want to study the relationship between the expenditure and the income with the following linear model: Expenditurei = β0 + β1Incomei + ǫi (i = 1, ..., 1000)

• 8

9 10 11 12 8.5 9.0 9.5 10.0 10.5 11.0 11.5 12.0 Income and Expenditure (data on log scale) Income Expenditure

Regression model Model estimation Properties OLS estimator

Example 1

The regression line has been estimated in R and it is estimated as

• Expenditurei = 2.8907 + 0.6947Incomei

(i = 1, ..., 1000) How to interpret these coefficients?

• β0:2.8907 is the average expenditure for a subject i with null

income;

• β1: increasing the income of 1 unit, the average expenditure

increases of 0.6947. More formally

The parameter β1 measures the expected change in yi if xi changes with one unit.

Regression model Model estimation Properties OLS estimator

Example 1

Suppose we want to predict the expenditure of a family with Income equal to 3. According to our model, the predicted value is 2.8907 + 0.6947 × 3 = 4.9748 And what is the predicted expenditure of a family with Income equal to 0?

Regression model Model estimation Properties OLS estimator

Example 1

Suppose we want to predict the expenditure of a family with Income equal to 3. According to our model, the predicted value is 2.8907 + 0.6947 × 3 = 4.9748 And what is the predicted expenditure of a family with Income equal to 0?

Regression model Model estimation Properties OLS estimator

Example 1

Suppose we want to predict the expenditure of a family with Income equal to 3. According to our model, the predicted value is 2.8907 + 0.6947 × 3 = 4.9748 And what is the predicted expenditure of a family with Income equal to 0?

Regression model Model estimation Properties OLS estimator

Example 2: Wages data

Wages data2: a sample (n = 3294) of individual wages with background characteristics like gender, race and years of schooling.

We want to study the relationship between the wages and gender. So we have: y1, ..., y3294: wages; x1, .., x3294: 0 − 1 variable denoting whether the individual is male (xi = 1) or female (xi = 0). dummy variable

2Chapter 2, Verbeek’s book

Regression model Model estimation Properties OLS estimator

Example 2: Wages data

Using the OLS approach the result is yi = 5.15 + 1.17xi It means that: The expected wage rate of a woman is 5.15 + 1.17 × 0 = 5.15; The expected wage rate of a man is 5.15 + 1.17 × 1 = 6.32 More formally we can say that

since x is a dummy variable, the parameter β1 is the expected wage differential between an arbitrary male and female.

Regression model Model estimation Properties OLS estimator

Multiple regression model

Suppose we have k explanatory variables. The model is yi = β0 + β1xi1 + β2xi2 + ...βkxik + ǫi Econometricians make frequent use of the following matrix notation: X =     1 x11 ... x1k 1 x21 ... x2k ... ... ... ... 1 xn1 ... xnk     y =     y1 y2 ... yn     β =     β0 β1 ... βk    

Regression model Model estimation Properties OLS estimator

Multiple regression model

In the matrix notation X is a n × (k + 1) matrix; y is a n × 1 vector; β is a (k + 1) × 1 vector. Using this notation, the regression model can be written as

y = Xβ + ǫ

where ǫ is a n × 1 vector of errors.

Regression model Model estimation Properties OLS estimator

Multiple regression model

The OLS estimators can be obtained by minimizing the expression: S(β) = (y − Xβ)

′(y − Xβ)

from which the least squares solution follows from differentiating with respect to β and setting the results to zero: dS(β dβ = −2(X ′y − X ′Xβ) = 0 Solving it gives the OLS solution

• β = (X ′X)−1X ′y = (

n

• i=1

xix′

i )−1 n

• i=1

xiyi

Regression model Model estimation Properties OLS estimator

Multiple regression model

The OLS estimators can be obtained by minimizing the expression: S(β) = (y − Xβ)

′(y − Xβ)

from which the least squares solution follows from differentiating with respect to β and setting the results to zero: dS(β dβ = −2(X ′y − X ′Xβ) = 0 Solving it gives the OLS solution

• β = (X ′X)−1X ′y = (

n

• i=1

xix′

i )−1 n

• i=1

xiyi

Regression model Model estimation Properties OLS estimator

Multiple regression model

The predicted value for y is given by

• y = X

β = X(X ′X)−1X ′y = Hy In linear algebra, the matrix H = X(X ′X)−1X ′ is known as projection matrix. It projects the vector y upon the columns of X. The matrix H is also known as hat matrix because it transforms y to y.

Regression model Model estimation Properties OLS estimator

Example 2: Wages data

Consider the wages data. We can extend our regression model with addition explanatory variables, such as the years of schooling (schooli), the experience in years (experi). The model is yi = β0 + β1malei + β2schooli + β3experi + ǫi

The model is now interpreted to describe the conditional expected wages of an individual given his or her gender, years

• f schooling and experience.

Regression model Model estimation Properties OLS estimator

Example 2: Wages data

The model is estimated as follows:

• yi = −3.38 + 1.34malei + 0.64schooli + 0.12experi

The coefficient of male measures the expected wage between male and female with the same schooling and experience: it means that if we compare an arbitrary male and female with the same schooling and same experience, the expected wage differential is 1.34

Regression model Model estimation Properties OLS estimator

Example 2: Wages data

The coefficient of school measures the expected wage difference between two individuals with the same experience, the same gender where one has one additional year of schooling and for the parameter of the experience?

In general, the coefficients in a multiple regression model can

• nly be interpreted under a ceteris paribus condition, which

says that the other variables that are included in the model are constant.

Regression model Model estimation Properties OLS estimator

Example 2: Wages data

The coefficient of school measures the expected wage difference between two individuals with the same experience, the same gender where one has one additional year of schooling and for the parameter of the experience?

In general, the coefficients in a multiple regression model can

• nly be interpreted under a ceteris paribus condition, which

says that the other variables that are included in the model are constant.

Regression model Model estimation Properties OLS estimator

Example 3: Salary data

Suppose we want to run a regression to find out if the average annual salary (yi) of public school teachers differs among three geographical regions in Country A with 51 states: (1) North (21 states) (2) South (17 states) (3) West (13 states). We estimate the following model yi = a0 + a1Statei + ǫi However, the variable State is a categorical variable with 3 possible

• utcomes (North, South and West).

Regression model Model estimation Properties OLS estimator

Example 3: Salary data

Suppose we want to run a regression to find out if the average annual salary (yi) of public school teachers differs among three geographical regions in Country A with 51 states: (1) North (21 states) (2) South (17 states) (3) West (13 states). We estimate the following model yi = a0 + a1Statei + ǫi However, the variable State is a categorical variable with 3 possible

• utcomes (North, South and West).

Regression model Model estimation Properties OLS estimator

Example 3: Salary data

The model is rewritten as follows: yi = a0 + a1D1i + a2D2i + ǫi where D1i and D2i are dummy variables defined as D1i = 1 if state i is in the North and D2i = 0 otherwise; D2i = 1 if state i is in the South and D2i = 0 otherwise; Clearly, when D1i = and D2i = 0, the state i is the West.

Regression model Model estimation Properties OLS estimator

Example 3: Salary data

In this model, we have only qualitative regressors, taking the value of 1 if the observation belongs to a specific category and 0 if it belongs to any other category. This makes it an ANOVA model.

How to interpret the coefficients?

Regression model Model estimation Properties OLS estimator

Example 3: Salary data

In this model, we have only qualitative regressors, taking the value of 1 if the observation belongs to a specific category and 0 if it belongs to any other category. This makes it an ANOVA model.

How to interpret the coefficients?

Regression model Model estimation Properties OLS estimator

Example 3: Salary data

Mean salary of public school teachers in the North Region (D1i = 1 and D2i = 0) is: E[Yi|D1i = 1, D2i = 0] = a0 + a1 Mean salary of public school teachers in the South Region (D1i = 0 and D2i = 1) is: E[Yi|D1i = 0, D2i = 1] = a0 + a2 Mean salary of public school teachers in the West Region (D1i = 0 and D2i = 0) is: E[Yi|D1i = 0, D2i = 0] = a0

Regression model Model estimation Properties OLS estimator

Example 3: Salary data

The mean salary of public school teachers in the West is equal to the intercept term a0 in the multiple regression equation and the differential intercept coefficients, a1 and a2, explain by how much the mean salaries of teachers in the North and South Regions vary from that of the teachers in the West. Thus, the mean salaries of teachers in the North and South is compared against the mean salary of the teachers in the West. Hence, the West Region becomes the base group or the benchmark group,i.e., the group against which the comparisons are made. The

• mitted category, i.e., the category to which no dummy is assigned,

is taken as the base group category also called corner point variable.

Regression model Model estimation Properties OLS estimator

Example 3: Salary data

Suppose the estimated model is

• Yi = 26158.62 − 1734.373D1i − 3264.615D2i

the mean salary of the teachers in the West (base group) is about 26158; the salary of the teachers in the North is lower by about 1734 than the salary in the West (that is 26158.62 - 1734.373=24424.14 is the average salary in the North); the salary of the teachers in the South is lower by about 3265 than the salary in the West (that is 26158.62 - 3264.615=22894 is the average salary in the South).

Regression model Model estimation Properties OLS estimator

Example 4: Salary data (plus)

Suppose we want to study the variation of the salary with respect to the State region (qualitative variable) and with respect to State Government expenditure on public schools xi (quantitative variable). The model is yi = a0 + a1D1i + a2D2i + a3xi + ǫi

Regression model Model estimation Properties OLS estimator

Example 4: Salary data (plus)

A regression model that contains a mixture of both quantitative as well as qualitative variables is called an Analysis

• f Covariance (ANCOVA) model. ANCOVA models are

extensions of ANOVA models. They are capable of statistically controlling the effects of quantitative explanatory variables (also called covariates or control variables), in a model that involves quantitative as well as qualitative regressors.

Regression model Model estimation Properties OLS estimator

Example 4: Salary data (plus)

The estimated model is

• Yi = 13269.11 − 1673.514D1i − 1144.157D2i + 3.2889Xi

For every 1$ increase in the State expenditure on public schools, a public school teacher’s average salary goes up by about 3.29 dollars; Meanwhile, for a state in the North region, the mean salary of the teachers is lower than that of West region by about 1673$ and for a state in the South region, the mean salary of teachers is lower than that of the West region by about 1144$. This means that the mean salary of the teachers in the North is about 11595$ (13269.11- 1673.514) and that of the teachers in the South is about 12125$ (13269.11- 1144.157. The mean salary of the teachers in the West Region (base category) is about 13269$ (intercept term).

Regression model Model estimation Properties OLS estimator

Example 4: Salary data (plus)

The estimated model is

• Yi = 13269.11 − 1673.514D1i − 1144.157D2i + 3.2889Xi

For every 1$ increase in the State expenditure on public schools, a public school teacher’s average salary goes up by about 3.29 dollars; Meanwhile, for a state in the North region, the mean salary of the teachers is lower than that of West region by about 1673$ and for a state in the South region, the mean salary of teachers is lower than that of the West region by about 1144$. This means that the mean salary of the teachers in the North is about 11595$ (13269.11- 1673.514) and that of the teachers in the South is about 12125$ (13269.11- 1144.157. The mean salary of the teachers in the West Region (base category) is about 13269$ (intercept term).

Regression model Model estimation Properties OLS estimator

The Gauss-Markov assumptions

Whether or not the OLS estimator β provides a good approximation of β depends crucially upon the assumptions that are made about the distribution of ǫi and its relation with xi.

A standard case in which the OLS estimators have good properties is characterized by the Gauss-Markov conditions.

Regression model Model estimation Properties OLS estimator

The Gauss-Markov assumptions

Whether or not the OLS estimator β provides a good approximation of β depends crucially upon the assumptions that are made about the distribution of ǫi and its relation with xi.

A standard case in which the OLS estimators have good properties is characterized by the Gauss-Markov conditions.

Regression model Model estimation Properties OLS estimator

The Gauss-Markov assumptions

For a linear regression model Y = Xβ + ǫ the Gauss-Markov conditions are E[ǫi] = 0 ǫ1, ..., ǫn and x1, ..., xn are independent; V (ǫi) = σ2; cov(ǫi, ǫj) = 0 for all i = j.

Regression model Model estimation Properties OLS estimator

The Gauss-Markov assumptions

E[ǫi] = 0 on average, the regression line should be correct. V (ǫi) = σ2 homoschedasticity cov(ǫi, ǫj) = 0 no autocorrelation. Taken together, these conditions imply that the error terms are uncorrelated drawings from a distribution with 0 mean and constant variance σ2. Using the matrix notation

E[ǫ] = 0 and V (ǫ) = σ2In

Regression model Model estimation Properties OLS estimator

The Gauss-Markov assumptions

E[ǫi] = 0 on average, the regression line should be correct. V (ǫi) = σ2 homoschedasticity cov(ǫi, ǫj) = 0 no autocorrelation. Taken together, these conditions imply that the error terms are uncorrelated drawings from a distribution with 0 mean and constant variance σ2. Using the matrix notation

E[ǫ] = 0 and V (ǫ) = σ2In

Regression model Model estimation Properties OLS estimator

Properties of the OLS estimator

1 The OLS estimator

β is unbiased (proof);

2 The OLS estimator is the best linear unbiased estimator

(Gauss-Markov theorem) and its variance is V ( β) = σ2(

n

• i=1

xix′

i )−1

Regression model Model estimation Properties OLS estimator

Properties of the OLS estimator

1 The OLS estimator

β is unbiased (proof);

2 The OLS estimator is the best linear unbiased estimator

(Gauss-Markov theorem) and its variance is V ( β) = σ2(

n

• i=1

xix′

i )−1

Regression model Model estimation Properties OLS estimator

Properties of the OLS estimator

To estimate the variance of β, we need to replace the unknown error variance σ2 with an estimate. An unbiased estimator of σ2 is s2 = 1 n − k

n

• i=1

e2

i =

1 n − k

n

• i=1

( yi − yi)2 where n is the number of observations and k the number of regressors in the model (including the intercept).

Regression model Model estimation Properties OLS estimator

Properties of the OLS estimator

Hence the variance of β can be estimated as

• V (

β) = s2(

n

• i=1

xix′

i )−1

We define standard error of β the quantity SE( β) =

• V (

β) The standard error of the estimator β is a measure for the accuracy

• f the estimator. We will define

se(βk) = s√ckk where ckk is the (k, k) element in (

i=1 xix′ i ).

Regression model Model estimation Properties OLS estimator

Properties of the OLS estimator

Hence the variance of β can be estimated as

• V (

β) = s2(

n

• i=1

xix′

i )−1

We define standard error of β the quantity SE( β) =

• V (

β) The standard error of the estimator β is a measure for the accuracy

• f the estimator. We will define

se(βk) = s√ckk where ckk is the (k, k) element in (

i=1 xix′ i ).

Regression model Model estimation Properties OLS estimator

One more assumption

The Gauss-Markov assumptionts state that the error term ǫi are mutually uncorrelated, are independent on X, have 0 mean and a constant variance but do not specify the shape of the distribution. For exact statistical inference, explicit distributional assumptions have to be made. The most widely used assumption is ǫ ∼ N(0, σ2In) Under this assumption it follows that Y ∼ N(Xβ, σ2In) and

• β ∼ N(β, σ2(X ′X)−1)

Regression model Model estimation Properties OLS estimator

One more assumption

The Gauss-Markov assumptionts state that the error term ǫi are mutually uncorrelated, are independent on X, have 0 mean and a constant variance but do not specify the shape of the distribution. For exact statistical inference, explicit distributional assumptions have to be made. The most widely used assumption is ǫ ∼ N(0, σ2In) Under this assumption it follows that Y ∼ N(Xβ, σ2In) and

• β ∼ N(β, σ2(X ′X)−1)