Advanced Mathematical Methods Part II Statistics Generalised - - PowerPoint PPT Presentation

1

Advanced Mathematical Methods

Part II – Statistics Generalised Linear Model

Mel Slater

http://www.cs.ucl.ac.uk/staff/m.slater/Teaching/Statistics/

2

Outline

Introduction The General Linear Model Least Squares Estimation Hypothesis Testing Analysis of Variance Multiple Correlation

3

Statistical Relationship

Experiments are usually conducted

to understand the relationship between a response variable y and a set of independent variables x1,x2,…,xk.

y is a random variable and the x’s

are thought of as constants.

E(y) = f(x1,x2,…,xk)

4

Linear Model

In practice the model is ‘linear’

• The linearity refers to linearity in the

parameters (not in the x’s)

We have observations on n individuals,

and, another way to write this is:

5

Matrix Representation

y = Xβ + ε

• A more succinct form
• y is an n*1 vector
• X is a n*p matrix (of constants)
• β is a p*1 vector
• ε is an n*1 vector of random variables

Note that p=k+1 if a constant term β0 is

included and the first column of X consists of 1s.

Normally there should be a constant term.

6

Problems

To estimate the unknown parameters β To make inferences about β In particular we can find confidence

intervals for β

We can test hypotheses, in particular the

hypotheses

• H0: β1= β2= …= βk = 0

(null hypothesis)

• H1: at least one βj ≠0

– Tests for relationship between y and X.

7

Least Squares Solution

β* = (XTX)-1XTy

• This is the L.S. solution
• Minimises the sum of squares of errors

between the fitted values and the true values of y.

E(β*) = β Var(β*) = σ2 (XTX)-1

• Where Var(ε) = σ2 I

8

Analysis of Variance

The total variation in the

response variable is

• It is the sample variance

without dividing by n-1

Let y* = X β*

• This is the fitted or predicted

response

Then the total variation in

the fitted variable is

2 1

) ( y y TSS

n i i −

= ∑

= 2 1

) * ( y y FSS

n i i −

= ∑

=

9

Analysis of Variance

The residual SS is defined as:

• RSS = TSS - FSS
• It is what is ‘unexplained’ by the model

If the model fitted the data then

• FSS = TSS and RSS = 0

10

Analysis of Variance

Now we make the further assumption that

ε ~ N(0, σ2 I)

Then under this assumption and the null

hypothesis:

• FSS / σ2 ~ Chi-squared (k)
• RSS /σ2 ~ Chi-squared (n-k)
• And MFSS and MRSS are independent

F = FSS/RSS ~ F(k,n-k)

• Large F should reject the null hypothesis

11

Analysis of Variance Table

Total n-1 Total Res/(n-k) Residual= deviance n-k Residual MFSS / MRSS Fitted/k Fitted k X vars F- Ratio MSS SS df Source

12

Multiple Correlation

R2 = Fitted SS/ Total SS This is the multiple correlation coefficient It is the proportion of the variation in the

response variable that is explained by the model.

R2 is between 0 and 1 It should be used together with the F-

Ratio to determine significance of the model

13

Testing individual β

Each β* ~ t-distribution on n-k degrees of

freedom, on the null hypothesis that β=0

This can be used to construct confidence

intervals or tests of significance

An approx rule is

• if β* / SE(β*) >2 reject null hypothesis

The ‘standard deviation’ for an estimate is

• ften called the ‘standard error’ (SE).

14

Estimating σ2

An unbiased estimator for σ2 is s2

• s2 = MRSS (mean residual SS)

Therefore

• SE-squared(β*) = s2(XTX)-1

15

Using GLIM

$units 24

!the number of obs

$data x1 x2 x3 y !variables $read !data follows this in logical row order !data goes here $finish !marks the end of the file Suppose the file name is file.txt

16

Using GLIM

$input 10 132 !reads in the file with

maximum field width of 132 chars

• Yes this is a very old system!!!!

$yvar y !declare which variable is the

response

$fit x1+x2+x3 !will fit the regression

model

17

Using GLIM

GLIM will print out the deviance and

degrees of freedom

• Deviance = residual sum of squares of the

model

• D.f. = degrees of freedom of the residual

Note if you fit the empty model, it will just

fit a constant term:

• $fit $ !fits the model y = beta0

The deviance for this is the Total SS

18

Using GLIM

$display e !will print out the estimates of

beta and their standard errors

This can be used to look at each beta

individually and assess its utility

The higher the ratio

• estimate/SE

the better that parameter and the more

that the corresponding x contributes to y.

19

Using GLIM

You can incrementally fit variables

• $fit +x4 $!adds x4 to the model

$fit . $ !refits the current model $display m $!displays the current model The advantage of GLIM compared to

MATLAB is that you don’t need to specify explicitly the X matrix

The ‘user interface’ is the disadvantage

20

Comparing Two Models

Suppose you have fitted a model

• $fit x1+x2+x3 !model S1

You want to see if adding more

terms makes a significant difference eg

• $fit +x4+x5 !model S2

Is S2 better than S1?

21

Comparing Two Models

Take the F-Ratio If this is large then reject the null

hypothesis that the additional variables make no difference.

)) 2 ( , ( ~ ) 2 ( / S df df F S MRSS df deviance F ∆ ∆ ∆ =