[PPT] - Marcel Dettling Marcel Dettling Institute for Data Analysis and d PowerPoint Presentation

SLIDE 1

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

Marcel Dettling Marcel Dettling

Institute for Data Analysis and Zurich University of Applied S

marcel.dettling@zhaw.ch htt // t t th h/ d ttli http://stat.ethz.ch/~dettling

ETH Zürich, November 29

Marcel Dettling, Zurich University of Applied Sciences

ression ression

d Process Design Sciences

9, 2010

1

SLIDE 2

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

L i ti R i Logistic Regression

has a Bernoulli d

{ }

0,1

i

Y ∈

The parameter of this distri

{ }

,

i

Now please note that:

( 1) [ ] P Y E Y

the most powerful notion o

( 1) [ ]

i i i

p P Y E Y = = =

see it as a model where w expected value of and th

i

Y Important:

1 1

...

i i

p x β β β = + + +

Marcel Dettling, Zurich University of Applied Sciences

ression ression

M d l Model

distribution. bution is , the success rate

i

p

f the logistic regression model is to

e try to find a relation between the he predictors! is no good here!

ip

x β

2

SLIDE 3

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

E l Example

Survival in Prem

35 30 age 25 2.8 2.9 20

Marcel Dettling, Zurich University of Applied Sciences

log10(we

ression ression

mature Birth

3.0 3.1

3

eight)

SLIDE 4

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

I f ith GLM Inference with GLMs

There are three tests that can

Goodness-of-fit test

based on comparing agai

based on comparing agai
not suitable for non-group
Comparing two nested m
likelihood ratio test leads
test statistics has an asym
Global test
Global test
comparing versus an emp
this is a nested model tak

Marcel Dettling, Zurich University of Applied Sciences

this is a nested model, tak

ression ression

be done: nst the saturated model nst the saturated model ped, binary data models to deviance differences mptotic Chi-Square distribution pty model with only an intercept ke the null deviance

4

ke the null deviance

SLIDE 5

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

N ll D i Null Deviance

Smallest model:

The smallest model is with

Fitted values will all be eq

Fitted values will all be eq
Our best fit (F) and the sm

A global test:

( ) ( )

(0) ( ) ( )

ˆ 2

F F

l l D y p D − = − Example:

( ) ( )

2 , l l D y p D − = −

Null deviance: 319.28 o Residual deviance: 235 9

Marcel Dettling, Zurich University of Applied Sciences

Residual deviance: 235.9

ression ression

hout predictors, only with intercept qual to ˆ

π

qual to mallest model (0) are nested

π

( )

(0)

ˆ D y p

( )

, D y p

n 246 degrees of freedom

94

n 244

degrees of freedom

5

94 on 244 degrees of freedom

SLIDE 6

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

M d l Di ti Model Diagnostics

Diagnostics are: g

as important with logistic re

linear regression models linear regression models

again based on differences

we now have to take into a equal for the different insta equal for the different insta we have to come up with n Pearson and Deviance res

Marcel Dettling, Zurich University of Applied Sciences

ression ression

egression as they are with multiple s between fitted & observed values ccount that the variances are not nces nces.

vel types of residuals:

siduals

6

SLIDE 7

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

P R id l Pearson Residuals

Take the difference between o by an estimate of the standard

ˆ ˆ ˆ (1 )

i i i i i

y p R p p − = −

is the contribution of the

2 i

R

( )

i i

p p

statistic for model compari It is important to note that It is important to note that value of two in absolute va

Marcel Dettling, Zurich University of Applied Sciences

ression ression

bserved an fitted value and divide

d deviation: e ith observation to the Pearson son. Pearson residuals exceeding a Pearson residuals exceeding a alue warrant a closer look

7

SLIDE 8

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

D i R id l Deviance Residuals

Take the contribution of the ith i.e. the chi-square statistic for

( )

(

( )

(

ˆ 2 log (1

i i i

d y p = − ⋅ + −

For obtaining a well interpreta root and the sign of the differe

ˆ ( )

i i i i

D sign y p d = − ⋅

It is important to note that value of two in absolute va

Marcel Dettling, Zurich University of Applied Sciences

ression ression

h observation to the log-likelihood, g , model comparison.

( )) ( ))

ˆ )log 1

i i

y p −

able residual, we take the square ence between true and fitted value: Pearson residuals exceeding a alue warrant a closer look

8

SLIDE 9

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

T k A b Pl Tukey-Anscombe Plo

Remark: sometimes studentiz

Tukey-Anscombe Plot 1

1 2 s

1

rson residual

3
2

Pear 0.2 0.4 0.6 0.8 1 fitted probabilities

Marcel Dettling, Zurich University of Applied Sciences

ression ression

t

t

zed residuals are used!

Tukey-Anscombe Plot 2

1 2 s

1

rson residual

3
2

Pear .0

3
2
1

1 2 3 linear predictor

9

SLIDE 10

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

T k A b Pl Tukey-Anscombe Plo

The Tukey-Anscombe plots in y p

xx <- predict(fit, type="re id l (fi " yy <- residuals(fit, type=" scatter.smooth(xx, yy, fami bli (h 0 lt 3) abline(h=0, lty=3)

Reasons:

using a non-robust smoothe
different types of residuals
different types of residuals
n the x-axis: probs or linea

Marcel Dettling, Zurich University of Applied Sciences

ression ression

t

t

n R are not perfect. Better use: p

esponse") " ") "pearson") ily="gaussian", pch=20)

er is a must can be used can be used ar predictor

10

SLIDE 11

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

M Di ti More Diagnostics

2 Residuals vs Lev 1 n resid.

2
1
Std. Pearson

4

4
3

S Cook's distance

68

0.00 0.02 0.04 Leverage

Marcel Dettling, Zurich University of Applied Sciences

glm(survival ~ I(log10(w

ression ression

verage

165 0.5

0.06 0.08 e

11

weight)) + age)

SLIDE 12

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

Bi i l R i Binomial Regression

Concentration in log of mg/l Number of insects n i in log of mg/l insects n_i 0.96 1.33 1.63 2.04 2 32

for the number of killed inse

2.32

we are mainly interested in these are grouped data: the a given predictor setting

Marcel Dettling, Zurich University of Applied Sciences

ression ression

M d l n Models

Number of killed insects y i killed insects y_i 50 6 48 16 46 24 49 42 50 44

ects, we have

50 44

~ ( , )

i i i

Y Bin n p

the proportion of insects surviving

i i i

ere is more than 1 observation for

12

SLIDE 13

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

M d l d E ti ti Model and Estimation

The goal is to find a relation: g

1

( 1| ,..., ) ~

i i p

p P Y x x = =

We will again use the logit link

p ⎛ ⎞

1 1

log 1

i i i

p x p β β ⎛ ⎞ = + ⎜ ⎟ − ⎝ ⎠

Here, is the expected value here fits within the GLM frame

i

p

( ) log

k i i i

n l n y y β ⎡ ⎛ ⎞ = + ⎢ ⎜ ⎟ ⎝ ⎠ ⎣

∑

Marcel Dettling, Zurich University of Applied Sciences

1 i i

y

=

⎝ ⎠ ⎣

∑

ression ression

n

1 1

...

i i p ip

x x η β β β = + + +

k function such that ( )

i i

g p η =

1

...

p ip

x β + +

e , and thus, also this model

ework. The log-likelihood is:

[ / ]

i i

E Y n

log( ) (1 )log(1 )

i i i i

p n y p ⎤ + − − ⎥ ⎦

13

⎦

SLIDE 14

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

Fitti ith R Fitting with R

We need to generate a two-co g contains the “successes” and > kill > killsurv killed surviv [1 ] 6 44 [1,] 6 44 [2,] 16 32 [3 ] 24 22 [3,] 24 22 [4,] 42 7 [5 ] 44 6 [5,] 44 6 > fit <- glm(killsurv~

Marcel Dettling, Zurich University of Applied Sciences

ression ression

lumn matrix where the first

the second contains the “failures” ~conc, family="binomial")

14

SLIDE 15

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

S O t t Summary Output

The result for the insecticide e

> summary(glm(killsurv ~ co Coefficients: E ti t Std E Estimate Std. E (Intercept) -4.8923 0. conc 3 1088 conc 3.1088 0.

Null deviance: 96 6881

Null deviance: 96.6881 Residual deviance: 1.4542 AIC: 24 675

Marcel Dettling, Zurich University of Applied Sciences

AIC: 24.675

ression ression

example is: p

nc, family = "binomial")

E l P (>| |) Error z value Pr(>|z|) .6426 -7.613 2.67e-14 *** 3879 8 015 1 11e 15 *** .3879 8.015 1.11e-15 ***

n 4

degrees of freedom

n 4 degrees of freedom
n 3 degrees of freedom

15

SLIDE 16

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

P ti f Kill d I Proportion of Killed I

I ti id P

1.0

Insecticide: Pro

0.6 0.8 ed insects 0.4

rtion of kill

0.0 0.2 Propo 0.5 1.0 C

Marcel Dettling, Zurich University of Applied Sciences

ression ression

I t Insects

ti f Kill d I t

portion of Killed Insects

1.5 2.0 2.5 Concentration

16

SLIDE 17

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

I f ith GLM Inference with GLMs

There are three tests that can

Goodness-of-fit test

based on comparing agai

based on comparing agai
not suitable for non-group
Comparing two nested m
likelihood ratio test leads
test statistics has an asym
Global test
Global test
comparing versus an emp
this is a nested model tak

Marcel Dettling, Zurich University of Applied Sciences

this is a nested model, tak

ression ression

be done: nst the saturated model nst the saturated model ped, binary data models to deviance differences mptotic Chi-Square distribution pty model with only an intercept ke the null deviance

17

ke the null deviance

SLIDE 18

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

G d f Fit T t Goodness-of-Fit Test

the residual deviance wil Paradigm: take twice the diffe for our current mod for our current mod the proportions pe

ˆ ( , ) 2 log ( ˆ

k i i

y D y p y n y ⎡ ⎛ ⎞ = + ⎢ ⎜ ⎟ ⎝ ⎠ ⎣

∑

Because the saturated model

1 i i

y

=

⎝ ⎠ ⎣

Because the saturated model deviance measures how close

Marcel Dettling, Zurich University of Applied Sciences

ression ression

t

l be our goodness-of-fit measure! g erence between the log-likelihood del and the saturated one which fits del and the saturated one, which fits rfectly, i.e. ˆ

/

i i i

p y n =

( ) )log ˆ ( )

i i i i

n y n y n y ⎤ ⎛ ⎞ − − ⎥ ⎜ ⎟ ⎝ ⎠⎦

fits as well as any model can fit the

( )

i i

n y − ⎝ ⎠⎦

fits as well as any model can fit, the e our model comes to perfection.

18

SLIDE 19

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

E l ti f th T Evaluation of the Tes

Asymptotics: y p If is truly binomial and the approximately

distributed. T

i

Y n

2

χ

pp y

(# ) k

f predictors

− −

> pchisq(deviance(fit), df. [1] 0.69287

Quick and dirty: : model i

Deviance df

: model i

More exactly: check

Deviance df

2

df df ±

Marcel Dettling, Zurich University of Applied Sciences

nly apply this test if at leas

ression ression

t st

are large, the deviance is The degrees of freedom is:

i

n

g

1 −

.residual(fit), lower=FALSE)

s not worth much s not worth much.

df

19

st all

5

i

n ≥

SLIDE 20

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

O di i Overdispersion

What if ??

Deviance df

1) Check the structural form

f

model diagnostics
predictor transformations

2) Outliers

should be apparent from

3) IID assumption for with

p

3) IID assumption for with

unrecorded predictors or

i

p

Marcel Dettling, Zurich University of Applied Sciences

subjects influence other s

ression ression

?? m of the model s, interactions, … the diagnostic plots hin a group hin a group inhomogeneous population

20

subjects under study

SLIDE 21

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

O di i R Overdispersion: a Re

We can deal with overdispers p

2

( 1 ˆ ˆ

n i

y X n p n p n p φ − = = ⋅∑ This is the sum of squared Pe

1 i i i

n p n p n p

=

− −

∑

This is the sum of squared Pe Implications:

regression coefficients rema
standard errors will be differe
standard errors will be differe
need to use an F-test for com

Marcel Dettling, Zurich University of Applied Sciences

ression ression

d emedy

ion by estimating: y g

2

ˆ ) ˆ (1 )

i i

n p p − earson residuals divided with the df (1 )

i

p − earson residuals divided with the df ain unchanged ent: inference! ent: inference! mparing nested models

21

SLIDE 22

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

R lt h C Results when Correc

> phi <- sum(resid(fit)^2)/ p ( ( ) )/ > phi [1] 0.4847485 > summary(fit, dispersion=p Estimate Std. E (Intercept) -4.8923 0. conc 3.1088 0.

(Dispersion parameter taken

Null deviance: 96.6881 Residual deviance: 1.4542

Marcel Dettling, Zurich University of Applied Sciences

AIC: 24.675

ression ression

ti O di i cting Overdispersion

/df.residual(fit) / ( ) phi) Error z value Pr(>|z|) .4474 -10.94 <2e-16 *** .2701 11.51 <2e-16 *** n to be 0.4847485)

n 4 degrees of freedom
n 3 degrees of freedom

22

SLIDE 23

Applied Statistical Regr Applied Statistical Regr

HS 2010 – Week 10

P i R i Poisson-Regression

When to apply? pp y

Responses need to be cou

for bounded counts the b

for bounded counts, the b
for large numbers the nor
The use of Poisson regress
unknown population size
when the size of the popu

and the probability of “suc Methods:

Marcel Dettling, Zurich University of Applied Sciences

Very similar to Binomial regre

ression ression

nts binomial model can be useful binomial model can be useful mal approximation can serve sion is a must if: and small counts ulation is large and hard to come by, ccess”/ the counts are small.

23