Statistics, inference and ordinary least squares Frank Venmans - - PowerPoint PPT Presentation

Statistics, inference and ordinary least squares

Frank Venmans

Statistics

Conditional probability

• Consider 2 events:
• A: die shows 1,3 or 5 => P(A)=3/6
• B: die shows 3 or 6 =>P(B)=2/6
• A∩B : A and B occur: die shows 3 =>P(A&B)=1/6
• AUB : A or B occur: die shows 1,3, 5 or 6 =>P(AorB)=4/6
• Addition rule: P(AorB)=P(A)+P(B)-P(A&B) (~ venn diagram)
• 𝑄 𝐵 𝐶 =

𝑄 𝐵& 𝐶 𝑄 𝐶

(~ venn diagram)

• P(A|B): prob of event A given that B occurs=1/2
• P(B|A): prob of event B given that A occurs=1/3
• Bayes’ Law: 𝑄 𝐵&𝐶 = 𝑄(𝐵|𝐶)P(B)=P(B|A)P(A)
• Event can be any set of outcomes. Example
• A: Random draw from belgian population with income >30,000
• B: Random draw from Belgian population with education >12 years
• P(A|B)≠P(A)

2 1 5 3 6 4

Income>30,000 Education>12

Independence

• 2 events A and B:

𝑄 𝐵 𝐶 = 𝑄 𝐵 ⇔ 𝑄 𝐶 𝐵 = 𝑄 𝐶 ⇔ 𝐵 𝑏𝑜𝑒 𝐶 𝑏𝑠𝑓 𝑗𝑜𝑒𝑓𝑞𝑓𝑜𝑒𝑓𝑜𝑢

• Two variables X and Y

𝑔 𝑦|𝑧 = 𝑔 𝑦 ⇔ 𝑔 𝑧 𝑦 = 𝑔 𝑧 ⇔ x and y are independent

• X and Y are independent if the conditional distribution of X given Y is the

same as the unconditional distribution of X.

• Independent variables do not necessarily have a zero correlation.
• Example: height of my sun and Indian GDP are correlated (both affected by time)
• Dependent variables may have a zero correlation in exceptional cases.
• Example: selection bias may compensate a causal effect (see further)

Cumulative Distribution Function CDF Probability Density Function PDF

• Notation:
• Random variables X,Y: ex. Yearly earnings and level of eduction
• Discrete if earnings are multiples of 100€ and eduction in years
• ~Continuous if earnings are expressed un eurocent and education in seconds
• Specific values of random variables:
• a,b or x,y
• Cumulative Distrubtion Function:
• probability that X is smaller than or equal to a
• 𝐺 𝑏 = 𝑄 𝑌 ≤ 𝑏
• Probability Density Function
• For discrete variables: f(a)=P(X=a)
• For continuous variables
• 𝑔 𝑏 = 𝑒𝐺 𝑏

𝑒𝑏

⇔ 𝐺 𝑏 = 𝑔 𝑌 𝑒𝑌

𝑏 −∞

• Area under the pdf =1 because 𝐺 ∞ = 1

Joint Cumulative Distribution Function

• Assume Y Yearly earnings and X level of education
• 𝐺 𝑦, 𝑧 = 𝑄 𝑌 < 𝑦 &𝑍 < 𝑧

Density function

• Joint Density Function
• For discrete variables:𝑔 𝑦, 𝑧 = 𝑄 𝑌 = 𝑦&𝑍 = 𝑧
• Continuous variables: 𝑔 𝑦, 𝑧 = 𝜖2𝐺 𝑦,𝑧

𝜖𝑦𝜖𝑧

• Marginal Denstity Function
• Discrete variables 𝑔 𝑦 = 𝑄 𝑌 = 𝑦 disregarding y
• Continuous variables 𝑔 𝑦 =

𝑔 𝑦, 𝑧 𝑒𝑧

𝑧=∞ 𝑧=−∞

• (red and blue line)
• Conditional Density Function
• Discrete variables 𝑔 𝑦|𝑧 = 𝑄 𝑌 = 𝑦 |𝑍 = 𝑧
• 𝑔 𝑦|𝑧 = 𝑔 𝑦,𝑧

𝑔 𝑧

• (intersections through the joint density function)

Regression as a conditional density function

Expected value

• Unconditional expected value
• For a discrete random variable : 𝐹 𝑌 = ∑𝑦𝑗𝑄 𝑦𝑗 = 𝜈
• For a continuous random variable : 𝐹 𝑌 =

𝑦𝑔 𝑦 𝑒𝑦

∞ −∞

= 𝜈

• Conditional expected value (in finance many expectations are conditional on the

information set at time t)

• 𝐹 𝑌 𝑍 = 𝐹𝑍[𝑌]=∑𝑦𝑗𝑄 𝑦𝑗|𝑍
• 𝐹 𝑌|𝑍 =

𝑦𝑔 𝑦|𝑧 𝑒𝑦

∞ −∞

• Variance= 𝜏2 = 𝐹[ 𝑌 − 𝜈 2]
• Covariance between X and Y= 𝜏𝑌,𝑍 = 𝐹

𝑌 − 𝜈𝑌 Y − 𝜈𝑍

• Skewness= 𝐹

𝑌−𝜈 𝜏 3

• Kurtosis= 𝐹

𝑌−𝜈 𝜏 4

Normal distribution

• 𝑔 𝑦 =

1 𝜏 2𝜌 exp − 1 2 𝑦−𝜈 𝜏 2

• Notation 𝑌~𝑂(𝜈, 𝜏2)
• Skewness=0
• Kurtosis=3
• Jacques-Berra test for normality:

tests if skewness and kurtosis are close to 0 and 3.

• Any linear combination of normally distributed variables

(correlated or not) is normally distributed

• Central limit theorem: the probability distribution of a variable that is the sum of an

infinite number of independent random variables with any distribution will be normally distributed.

Chi square distribution

• 𝑍 = ∑

𝑌𝑗

2𝑥𝑗𝑢ℎ 𝑌𝑗~𝑂 0,1 𝑏𝑜𝑒 𝑏𝑚𝑚 𝑌𝑗𝑗𝑜𝑒𝑓𝑞𝑓𝑜𝑒𝑓𝑜𝑢 𝑜 𝑗=1

follows a 𝜓2distribution with n degrees of freedom.

• 𝑍~𝜓𝑜

2

Student t distribution

• 𝑎 =

𝑌

𝑍 𝑜

𝑥𝑗𝑢ℎ 𝑌~𝑂 0,1 𝑏𝑜𝑒 𝑍~𝜓𝑜

2 𝑏𝑜𝑒 𝑌 𝑗𝑜𝑒𝑓𝑞𝑓𝑜𝑒𝑓𝑜𝑢 𝑔𝑠𝑝𝑛 𝑍

follows a student or t-distribution with n degrees of freedom

• 𝑎~𝑢𝑜
• Higher variance and kurtosis than

the standardized normal distribution

• Converges to the normal distribution

for large n: 𝑢∞ = 𝑂 0,1

F distribution

• Z= X/n

Y/m with X~χ𝑜

2 𝑏𝑜𝑒 𝑍~𝜓𝑛 2 𝑏𝑜𝑒 𝑌 𝑗𝑜𝑒𝑓𝑞𝑓𝑜𝑒𝑓𝑜𝑢 𝑔𝑠𝑝𝑛 𝑍 follows

an F distribution with n and m degrees of freedom.

• 𝑎~𝐺

𝑜,𝑛

Inference

Statistical inference

• Try to say something about the real distribution of a random variable based
• n a sample.
• The real distribution corresponds to an infinitely repeated event (ex dice), the

entire population, entire set of possible ‘states of the world’ in a future period etc.

3 types of inference

• Point estimator:
• Ex: sample mean, sample variance, marginal effect in a linear regression (beta), correlation…
• Concept of repeated sampling: every sample gives another estimator 𝜄

=> 𝜄 will follow a prob distribution

• Unbiased: Expected value of estimator corresponds to the real parameter 𝐹 𝜄

= 𝜄

• Consistent: The estimator can get arbitrarily close to the real parameter by increasing the sample size

plim

𝑜→∞

𝜄 = 𝜄

• Ex: sample variance estimator 𝑡 ² =

1 𝑜 ∑ 𝑧𝑗 − 𝑧

2

𝑗

is a biased but consistent estimator of the variance

• Efficient estimator: 𝑤𝑏𝑠(𝜄

) is small

• Interval estimation:
• Ex: given the observed sample, the real mean lays between 1 and 3 with 95% probability
• Hypothesis testing:
• Ex: if the null hypothesis is true (𝜈 = 2), what is the probability of a random sample to have a more

extreme (less likely) outcome than the observed sample mean of 4 and sample variance of 2.

Example: Sample mean

• Income of Belgian households: a random variable following a distribution

with mean 𝜈 and variance 𝜏² (distribution is skewed, not normal)

• You have a sample of n individuals. You want to say something about 𝜈 and

𝜏²

• Estimator of 𝜈: sample mean y

=

𝑧1+𝑧2+𝑧3…𝑧𝑜 𝑜

• Estimator will be different each time you draw a different sample=>sample

mean will follow a distribution, which is different from the distribution of y.

• Central limit theorem =>the sample mean converges to a normal

distribution even if y does not follow a normal distribution.

Sample mean: variance known

• 𝑧

𝑏𝑡𝑡𝑧𝑛𝑞𝑢𝑝𝑢𝑗𝑑 ~N 𝜈, 𝜏2 𝑜

⇒

𝑧 −𝜈 𝜏 𝑜 𝑏𝑡𝑡𝑧𝑛𝑞𝑢𝑝𝑢𝑗𝑑 ~N(0,1)

• This allows to determine a 95%confidence interval

𝑄 −1,96 <

𝑧 −𝜈 𝜏/ 𝑜 < 1,96 = 0,95 ⇔ 𝑄 𝑧

− 1,96

𝜏 𝑜 < 𝜈 < y

+ 1,96

𝜏 𝑜 =0,95

• When interval includes zero we say that the sample mean is not

significantly different from zero at the 5% confidence level.

Sample mean: variance unknown and y normally distributed

• Both mean and variance will need to be estimated.
• Estimator for variance: 𝑡 2 =

1 𝑜−1 ∑

𝑧𝑗 − 𝑧 2

𝑜

• If Y follows a normal distribution⇔ (𝑜−1)𝑡 2

𝜏2

= ∑

𝑧𝑗−𝑧 𝜏 2 𝑜

~𝜓𝑜−1

2

(no proof but intuitive)

• 𝑧

−𝜈 𝑡 / 𝑜 =

𝑧 −𝜈 𝜏/ 𝑜 𝑡 𝜏

=

𝑧 −𝜈 𝜏/ 𝑜 𝑜−1 𝑡 2 (𝑜−1)𝜏2

~ 𝑂 0,1

𝜓𝑜−1 2 𝑜−1

= 𝑢𝑜−1

• This allows to determine a 95% confidence interval (ex. n=21)
• 𝑄 −2,086 < 𝑧

−𝜈

𝑡 𝑜

< 2,086 = 0,95 ⇔ 𝑄 𝑧 − 2,086 𝑡

𝑜 < 𝜈 < y

+ 2,086 𝑡

𝑜 =0,95

• For large n, the t distribution converges to the normal distribution

Hypothesis testing

• Null hypothesis 𝐼0: 𝜄 = 𝜄0

ex: 𝐼0: 𝜄 = 0

• One sided test 𝐼𝐵: 𝜄 > 𝜄0 (𝑝𝑠 𝜄 < 𝜄0)

ex: 𝐼𝐵: 𝜄 > 0

• Two sided test 𝐼𝐵: 𝜄 ≠ 𝜄0

ex: 𝐼

𝐵: 𝜄 ≠ 0

• 2 regions:
• If observed data (test statistic) falls in rejection region =>reject H0
• If observed data (test statistic) falls in acceptence region =>accept H0
• Imagine you have 10 months of data and you observe a mean monthly return
• f the stock of Apple of 0,8% and you want to test if this mean is different

from a zero return.

• Assume the standard error of the return is observed to be 1,58%, so the standard error
• f the mean is 1,58%

10 = 0,5%

One sided test vs 2-sided test

• One sided test: if the real mean was zero, what would

be the probability to observe an estimator larger than 0,8%(1,58%)?

• Standardize your outcome 𝐼0: 𝜈 = 0 ⇒

𝑧 −0 𝑡 / 𝑜 ~𝑢𝑜

• ⇒ 𝑄 𝑍 >

𝑧 −0

𝑡 𝑜

= 𝑄 𝑍 > 1,6 =0,05 (Pvalue given by Stata)

• Two sided test: if the real mean was zero, what would

be the probability that the sample mean was outside the interval of [−

𝑧 𝑡 / 𝑜, 𝑧 𝑡 / 𝑜]

• 𝐼0: 𝜈 = 0 ⇒ 1 − 𝑄(−

𝑧 𝑡 / 𝑜 < 𝑌 < 𝑧 𝑡 / 𝑜)

= 1- P(-1,6<X<1,6)=0,10 (Pvalue given by Stata)

• Remark: n is small so assumption of normally distributed

returns is needed

Type I and type II errors

Do not reject H0 Reject H0 H0 true Type I error, α (ex 5%) H0 false Type II error, β 1-β=power of test

• Level of significance: probability to reject the null if the null is true
• Power of the test: probability to reject the null if the nulle is false
• Reduce probability of type I error =>increase probability of Type II error
• More efficient estimator=>reduce probability of Type II error=increase power of the

test

• Increase sample size=>reduce probability of Type II error=increase power of the test
• General rule: go for a large sample, in small samples you may only see phenomenons

big as an elephant, that you knew allready before doing the test, all the rest has an insignificant effect.

Ordinary least squares

Regression

• Assume we want to know the relationship between sales and advertising expenditure
• OLS: minimize squared distance between points and regresssion line

Y=Sales X=advertising expenditure 𝜗𝑗 𝑍 = 𝛽 + 𝛾𝑌 Slope= β α 𝛽 + 𝛾𝑌𝑗 𝑍

𝑗

Population regression line vs sampling regression line

• Estimators and regression line (orange) will be different for each sample.
• Would you use a one-sided or two-sided t-test for beta?

Y=Sales 𝜗𝑗 𝑍 = 𝛽 + 𝛾𝑌 Slope= β α 𝛽 + 𝛾𝑌𝑗 𝑍

𝑗

𝜗 𝑗 𝑍 = 𝛽 + 𝛾 𝑌 Slope= 𝛾 𝛽 + 𝛾 𝑌 Advertising expenditures

Assumptions of OLS

• 5 Gauss-Markov assumptions:
• The true model is 𝑍 = 𝛽0 + 𝛾1𝑌1 + 𝛾2𝑌2 + ⋯ + 𝜗 with E 𝜗 = 0 (linearity)
• No perfect collinearity (you cannot write X1 as a linear combination of the other Xj’s)
• Homoscedastic errors 𝐹 𝜗𝑗

2 = 𝜏²

• Uncorrelated errors 𝐹 𝜗𝑗𝜗𝑘 = 0
• 𝐹 𝜗|𝑌1, 𝑌2 = 0 (exogenous explainatory variables, no endogeneïty)
• If 5 assumptions are met, OLS is Best Linear Unbiased Estimator (BLUE)
• If the errors are normally distributed, OLS is Best Unbiased Estimator (BUE)
• OLS with non-normal errors is still unbiased and consistent!
• 𝛾

follows a t-distribution only if errors are normal => be prudent with interpreting confidence intervals in small samples

𝑛𝑏𝑢𝑠𝑗𝑦 𝑜𝑝𝑢𝑏𝑢𝑗𝑝𝑜 𝐹 𝜗𝜗′ = 𝜏𝐽

The math behind OLS (optional, only for those who like it)

• Consider Matrix notation of model: 𝑍 = 𝑌𝛾 + 𝜗
• If there is an intercept, X contains a column of one’s
• Minimum distance estimator

min

𝛾 ∑ 𝜗𝑗 2 𝑜

= min

𝛾 𝜗′𝜗 = min 𝛾

𝑍 − 𝑌𝛾 ′ 𝑍 − 𝑌𝛾 = min

𝛾

𝑍′𝑍 − 2𝛾′ 𝑌′𝑍 + 𝛾′ 𝑌′𝑌𝛾 𝑔𝑗𝑠𝑡𝑢 𝑒𝑓𝑠𝑗𝑤𝑏𝑢𝑗𝑤𝑓: −2𝑌′𝑍 + 2𝑌′𝑌𝛾 = 0 ⇔ 𝛾 = 𝑌′𝑌 −1𝑌′𝑍

• Method of moments: errors must be uncorrelated with the regressors

X′𝜗 = 0 ⇔ 𝑌′ 𝑍 − 𝑌𝛾 = 0 ⇔ 𝛾 = 𝑌′𝑌 −1𝑌′𝑍

• Maximum likelihood under normal distribution of error term
• 𝑚𝑗𝑙𝑓𝑚𝑗ℎ𝑝𝑝𝑒 =

1 2𝜌𝜏2 exp − 𝜗𝑗

2

2𝜏2 𝑗

• 𝑀𝑝𝑕𝑚𝑗𝑙𝑓𝑚𝑗ℎ𝑝𝑝𝑒 = −

n 2 log 2𝜌𝜏2 − 1 2𝜏2 ∑ 𝜗𝑗² 𝑜

• Minimising the loglikelihood boils down to minimum distance estimator=>OLS is BUE

OLS inference

• If errors are normally distributed, the estimate 𝛾

follows a student t distribution (only assymptotically the case if errors are not normally distributed)

• If Errors are correlated or heteroscedastic, the variance of beta 𝜏

𝛾 2 can

be increased to take that into account (option ‘robust’ in stata)

• Stata command: regress Y X1 X2, robust
• Default includes a constant, you can add option noconstant
• Exogeneity of X’s cannot be tested. The problem of endogeneïty is

most important condition for causal interpretation of beta’s (see next week)

Avoid endogeneïty

• Conditions 1 and 5 imply 𝐹 𝑍 𝑌1, 𝑌2 = 𝛽0 + 𝛾1𝑌1 + 𝛾2𝑌2 .
• This allows a causal interpretation of beta’s: an increase of X1 by one

unit, all other relevant factors being equal, will have an effect 𝛾1 on Y.

• Intuition: ‘all other factors being equal’ implies that all factors that

drive the error term (and thus Y), are uncorrelated to the variables of interest X.

The effect of marketing « all else being equal »

Sales Marketing expenditures Error= all other factors Innovative company Competitors Quality of product Delivery time Business cycle

𝐹 𝜗|𝑌 ≠ 0 ⇒ 𝑑𝑝𝑤 𝜗, 𝑌 ≠ 0 ⇒ 𝜗 and X are driven by common factors.

Fixed effect panel regression to avoid endogeneïty

Sales Marketing expenditures Fixed effect= all factors that are constant over time Innovative company Competitors Quality of product Delivery time Idiosyncratic error= all other factors that change over time Business cycle

Fixed effects - random effect - pooled panel

• 3 ways of writing the same fixed effect model:
• Y= 𝑌𝛾

+ ∑ 𝛿𝑗 𝐸𝑗

𝑗

+ 𝜗 with Di a dummy variable for company i.

• 𝑍

𝑗𝑢 = 𝑌𝑗𝑢𝛾

+ 𝛿𝑗 + 𝜗it

• 𝑍

𝑗𝑢 − 𝑍 𝑗 = (𝑌𝑗𝑢−𝑌

𝑗)𝛾 + 𝜗𝑗𝑢 − 𝜗 𝑗 =‘within estimator’ (obtained by subtracting the sum of eq 2 over time periods)

• Beta measures the effect of a deviation from mean marketing expenditures on the deviation of mean sales ‘within’ a

company i. =>The difference in mean sales between a company with high and low average marketing expenditures does not drive the estimation of beta => be careful with measurement errors and lagged effects because part of variability is filtered out.

• If theory indicates that you may avoid a source of endogeneity and you have enough data to find significant effects =>

use fixed effects!

• Random Effect model and Pooled panel regression assume that none of the factors that drives a company specific

effect drives any of the X’s as well: fixed effects are uncorrelated with X’s

• Pooled panel regression is an OLS as if there was no panel structure: every observation has equal weight
• Random effect model is a Generalized Least Squares (GLS) estimator: the observed heterogeneity and serial

correlation is used to make estimator more efficient compared to the pooled panel regression

Alternative functional forms

• If X increases by 1, Y will increase by β
• 𝑍 = 𝛽 + 𝛾𝑌 + 𝜗
• 𝑒𝑍

𝑒𝑌 = 𝛾= marginal effect

• If X increases by 1%, Y will increase by β%
• 𝑚𝑜𝑍 = 𝛽 + 𝛾𝑚𝑜𝑌 + 𝜗 ⇔ 𝑍 = 𝑓𝛽𝑌𝛾𝑓𝜗
• 𝑒𝑚𝑜𝑍

𝑒𝑚𝑜𝑌 = 𝛾 = 𝑒𝑍/𝑍 𝑒𝑌/𝑌 = 𝑓𝑚𝑏𝑡𝑢𝑗𝑑𝑗𝑢𝑧

• If X increases by 1, Y will increase by β%
• 𝑚𝑜𝑍 = 𝛽 + 𝛾𝑌 + 𝜗 ⇔ 𝑍 = 𝑓𝛽𝑓𝛾𝑌𝑓𝜗
• 𝑒𝑚𝑜𝑍

𝑒𝑌 = 𝛾 = 𝑒𝑍/𝑍 𝑒𝑌 = 𝑕𝑝𝑥𝑢ℎ 𝑠𝑏𝑢𝑓 (think of X as time)

• If X increases by 1%, Y will increase by β
• 𝑍 = 𝛽 + 𝛾𝑚𝑜𝑌 + 𝜗 ⇔ 𝑓𝑍 = 𝑓𝛽𝑌𝛾𝑓𝜗
• 𝑒𝑍

𝑚𝑜𝑌 = 𝛾 = 𝑒𝑍

𝑒𝑌 𝑌

• Any transformation (lnX, 1/X, X², X³, expX) is in principle allowed
• Transformation can be justified by theory (in most cases) or by the data (see graphs)

Y Y Y X X X

Some useful commands in Stata

• Type help … in the command windows for the following commands:
• summarize: summarize information about a variable or dataset
• tabulate var1 var2 : tabulation table to explore data
• destring : define a variable as numerical if it would be imported as ‘string’ (text)
• generate var1=var2+var3 : generates a new variable (also for log transformation)
• replace var1=0 if var2==. & var3>36 :
• logical expressions with == ;
• dot= missing (or ∞) => if var3 is missing, it satisfies var3>36 !
• replace var1=1 if l.var2==25 | var3==var4 : lag operator only if dataset defined as

time series or panel.

• regress y x1 x2 : regression with many options
• tset year : define dataset to be a time series with year name of time variable

Commands in Stata:

• xtset i t : define a database to be a panel with i name of person or company
• xtreg : fixed effect or random effect panel regression
• Create an id per companyname:
• egen id= group(companyname)
• egen stands for ‘extensions to generate’, operates on groups of observations.
• generate only applies to one observation at a time
• Calculate mean (over time) of variable income per company id:
• by id: egen meanincome = mean(income)
• Eliminate extreme values beyond 99th percentile of variable income:
• Summarize income, detail
• Replace income=. if income>`r(p99)’
• Most commands create different macro’s (mentioned at the end of help document). You can

use them with `…’ . Since they are local macro’s, they are erased at some point (in this case until the next command that uses r to store results.