STAT 213 Logistic Regression II Colin Reimer Dawson Oberlin - - PowerPoint PPT Presentation

Outline Logistic Regression Fitting the Model Assessment and Testing

STAT 213 Logistic Regression II

Colin Reimer Dawson

Oberlin College

28 April 2016

Outline Logistic Regression Fitting the Model Assessment and Testing

Outline

Logistic Regression Fitting the Model Assessment and Testing

Outline Logistic Regression Fitting the Model Assessment and Testing

Reading Quiz (Multiple Choice)

Two logistic models have the same β0 but different β1. For each of the following, state whether the statement must be true, might be true, or cannot be true

• b. The graphs of log(odds) versus X cross the Y -axis at the

same value of y

• d. The graphs of P(Y = 1) versus X cross the line

P(Y = 1) = 0.5 at the same value of X

• e. The graphs of P(Y = 1) versus X the line x = 0.5 at the

same value of Y .

Outline Logistic Regression Fitting the Model Assessment and Testing

For Tuesday

• Write up: 9.14, 9.22, 9.26
• Read: 10.1-10.2
• Answer: 9.12, 10.2, 10.4
• Soon: Project 3 (due on the last day of classes)

Outline Logistic Regression Fitting the Model Assessment and Testing

Quantitative Vs. Categorical Predictor and Response

Response Quantitative Categorical Predictor Quantitative Linear Reg. Logistic Reg. Categorical ANOVA

Outline Logistic Regression Fitting the Model Assessment and Testing

Binary Logistic Regression

Response variable (Y ) is categorical with two categories (i.e., binary).

• Code Y as an indicator variable: 0 or 1
• Assume (for now) a single quantitative predictor, X

Outline Logistic Regression Fitting the Model Assessment and Testing

Two Equivalent Forms of Logistic Regression

Probability Form π = eβ0+β1X 1 + eβ0+β1X Logit Form log

• π

1 − π

• = β0 + β1X

π : Probability that Y = 1 π 1 − π : Odds that Y = 1 log

• π

1 − π

• : Log odds, or logit that Y = 1

Outline Logistic Regression Fitting the Model Assessment and Testing

Example: Golf Putts

Distance (ft) 3 4 5 6 7 # Made 84 88 61 61 44 # Missed 17 31 47 64 90 Total 101 119 108 125 134

• 1. Estimate the probability of success at each length
• 2. Estimate the odds of success at each length
• 3. Estimate the log odds of success at each length
• 4. Plot each of these against distance

Outline Logistic Regression Fitting the Model Assessment and Testing

Odds Ratios

Logit and Odds log

• π

1 − π

• = β0 + β1X

π 1 − π = eβ0+β1X

• In the model, for each 1 unit increase in X, the logit

increases by β1.

• Equivalently: For each 1 unit increase in X, the odds are

multiplied by eβ1

• In other words, eβ

1 is the odds ratio resulting from a one

unit change in X, with β1 the log odds ratio.

Outline Logistic Regression Fitting the Model Assessment and Testing

Odds Ratios

The odds ratio associated with a binary response Y at two different predictor values X = x2 vs. X = x2 is the ratio of the

• dds; that is:

Odds Ratio(x2 vs. x1) = π(x2)/(1 − π(x2)) π(x1)/(1 − π(x1)) We can estimate this from a sample using:

• Odds Ratio(x2 vs. x1) = ˆ

π(x2)/(1 − ˆ π(x2)) ˆ π(x1)/(1 − ˆ π(x1))

Outline Logistic Regression Fitting the Model Assessment and Testing

Example: Golf Putts

Distance (ft) 3 4 5 6 7 # Made 84 88 61 61 44 # Missed 17 31 47 64 90 Total 101 119 108 125 134 ˆ π 0.832 0.739 0.565 0.488 0.328 Odds 4.94 2.84 1.30 0.95 0.49 Log Odds 1.60 1.04 0.26

• 0.05
• 0.71
• 5. Find the sample odds ratio for success for 4 ft. vs. 3 ft; 5
• vs. 4; 6 vs. 5; 7 vs. 6
• 6. Take the log of each of these to get the (additive) change

in the logit. Should be slopes of lines “connecting the dots” (since ∆X = 1).

Outline Logistic Regression Fitting the Model Assessment and Testing

Example: Golf Putts

Distance (ft) 3 4 5 6 7 # Made 84 88 61 61 44 # Missed 17 31 47 64 90 Odds 4.94 2.84 1.30 0.95 0.49 Log Odds 1.60 1.04 0.26

• 0.05
• 0.71

OR 0.575 0.457 0.734 0.513 ∆ Log Odds

• 0.56
• 0.78
• 0.31
• 0.66
• In the data, successive ORs (changes in log odds) are

different

• The model fits a constant ratio (slope for log odds)
• 7. Draw a single line through your logit plot and get an

estimated slope and intercept. These are your ˆ β0 and ˆ β1.

Outline Logistic Regression Fitting the Model Assessment and Testing

Example: Golf Putts

library("mosaic") Putts <- data.frame(Distance = 3:7, Made = c(84,88,61,61,44), Total = c(101,119,108,125,134)) Putts <- mutate(Putts, PropMade = Made / Total) (model <- glm(PropMade ~ Distance, weights = Total, data = Putts, family = "binomial")) Call: glm(formula = PropMade ~ Distance, family = "binomial", data = Putts, weights = Total) Coefficients: (Intercept) Distance 3.2568

• 0.5661

Degrees of Freedom: 4 Total (i.e. Null); 3 Residual Null Deviance: 81.39 Residual Deviance: 1.069 AIC: 30.18

Outline Logistic Regression Fitting the Model Assessment and Testing

Example: Golf Putts (Probabilities)

xyplot(PropMade ~ Distance, data = Putts) f.hat <- makeFun(model) plotFun(f.hat(Distance) ~ Distance, add = TRUE) Distance PropMade

0.4 0.5 0.6 0.7 0.8 3 4 5 6 7

Outline Logistic Regression Fitting the Model Assessment and Testing

Example: Golf Putts (Odds)

f.hat <- makeFun(model, transform = function(p){p/(1-p)}) xyplot(PropMade/(1 - PropMade) ~ Distance, data = Putts) plotFun(f.hat(Distance) ~ Distance, add = TRUE) Distance PropMade/(1 − PropMade)

1 2 3 4 5 3 4 5 6 7

• exp(-0.5661) ## Odds ratio for a one foot increase in Distance

[1] 0.5677353

Outline Logistic Regression Fitting the Model Assessment and Testing

Example: Golf Putts (Log Odds)

f.hat <- makeFun(model, transform = logit) xyplot(logit(PropMade) ~ Distance, data = Putts) plotFun(f.hat(Distance) ~ Distance, add = TRUE) Distance logit(PropMade)

−0.5 0.0 0.5 1.0 1.5 3 4 5 6 7

• 0.5661 ## Log (odds ratio) / rate of change in log odds / slope of logit

Outline Logistic Regression Fitting the Model Assessment and Testing

Reconstructing Odds Ratio

• The logistic regression output from R gives us ˆ

β0 and ˆ β1. But unlike in linear regression, these are not very interpretable on their own.

• We have seen that β1 corresponds to “rate of change in

log odds”. Better to convert to “odds ratio” per unit change in X.

• What do we do to β1 to get this?

Outline Logistic Regression Fitting the Model Assessment and Testing

Choosing ˆ β0 and ˆ β1

Recall that in linear regression, we choose ˆ β0 and ˆ β1 to minimize RSS =

• i

(Yi − f(Xi))2 =

• i

(Yi − ˆ β0 − ˆ β1X)2 For a logistic model, choose ˆ β0 and ˆ β1 to maximize the probability of the data under the model. Pr(Data|Model) =

n

• i=1

ˆ πYi

i (1 − ˆ

πi)1−Yi =

n

• i=1
• e

ˆ β0+ˆ β1Xi

1 + eˆ

β0+ˆ β1Xi

Yi 1 1 + eˆ

β0+ˆ β1Xi

1−Yi

Outline Logistic Regression Fitting the Model Assessment and Testing

Maximum Likelihood

• Pr(Data|Model) is called the likelihood of the model.
• In fact, when we assume heteroskedastic Normal

residuals, the RSS is the negative log likelihood.

• So we’ve secretly been doing max likelihood this whole

time.

• But whereas MLE for Normal-linear model was a calculus

problem, MLE for logistic requires an iterative algorithm.

Outline Logistic Regression Fitting the Model Assessment and Testing

Conditions for Logistic Regression

• 1. Linearity (log odds depends linearly on X)
• 2. Independence (no clustering or time/space dependence)
• 3. Random (data comes from a random sample, or random

assignment)

• 4. Normality no longer applies! (Response is binary, so it can’t)
• 5. Homoskedasticity no longer required! (In fact, more variance

when ˆ π near 0.5)

Outline Logistic Regression Fitting the Model Assessment and Testing

Checking Linearity

• Can’t just transform response via logit to check linearity...
• Unless data is binned... then can take logit of proportion

per bin

Outline Logistic Regression Fitting the Model Assessment and Testing

Binned Data

xyplot(logit(PropMade) ~ Distance, data = Putts, type = c("p","r")) Distance logit(PropMade)

−0.5 0.0 0.5 1.0 1.5 3 4 5 6 7

• Logits are fairly linear

Outline Logistic Regression Fitting the Model Assessment and Testing

Equivalent Model Code for Binned Data

Putts <- mutate(Putts, Missed = Total - Made) (m2 <- glm(cbind(Made,Missed) ~ Distance, data = Putts, family = "binomial")) Call: glm(formula = cbind(Made, Missed) ~ Distance, family = "binomial", data = Putts) Coefficients: (Intercept) Distance 3.2568

• 0.5661

Degrees of Freedom: 4 Total (i.e. Null); 3 Residual Null Deviance: 81.39 Residual Deviance: 1.069 AIC: 30.18

Outline Logistic Regression Fitting the Model Assessment and Testing

Hypothesis Test for β1

In linear regression, we computed tobs = ˆ β1 − 0 ˆ se(ˆ β1) and found P-value = Pr(|Tn−2| ≥ |tobs|) In logistic regression we can use a Normal approximation: zobs = ˆ β1 − 0 ˆ se(ˆ β1) and get P-value = Pr(|Z| ≥ |zobs|)

Outline Logistic Regression Fitting the Model Assessment and Testing

In R

data("Election08") model <- glm(ObamaWin ~ BA, data = Election08, family = "binomial") summary(model) Call: glm(formula = ObamaWin ~ BA, family = "binomial", data = Election08) Deviance Residuals: Min 1Q Median 3Q Max

• 1.7130
• 0.8697

0.2436 0.7314 1.7952 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept)

• 9.4667

2.9656

• 3.192

0.00141 ** BA 0.3706 0.1143 3.243 0.00118 **

• Signif. codes:

0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 69.737

• n 50

degrees of freedom Residual deviance: 49.689

• n 49

degrees of freedom

Outline Logistic Regression Fitting the Model Assessment and Testing

Confidence Interval for β1

Same principle applies for confidence interval... CI(∆logit) : ˆ β1 ± z∗ · ˆ se( ˆ β1)

Estimate Std. Error z value Pr(>|z|) (Intercept) -9.4667396 2.9655946 -3.192189 0.001411987 BA 0.3705518 0.1142507 3.243321 0.001181449

But... β1 is the rate of change of the logit, which is hard to

• understand. More common to report a CI for the odds ratio.

CI(OR) : (eβ(lwr)

1

, eβ(upr)

1

)

Outline Logistic Regression Fitting the Model Assessment and Testing

Or, in R...

confint(model) 2.5 % 97.5 % (Intercept) -16.3335145 -4.4506636 BA 0.1789443 0.6366837 confint(model) %>% exp() 2.5 % 97.5 % (Intercept) 8.062037e-08 0.01167082 BA 1.195954e+00 1.89020201

Outline Logistic Regression Fitting the Model Assessment and Testing

Logistic Analogs of F-test, R2, etc.

• Rather than R2, we can use the residual deviance to

measure lack of fit (so, smaller is better) Deviance(Model) = −2 log(likelihood(Model)) Residual Deviance = Deviance(Fitted Model) Null Deviance = Deviance(Null Model)

Outline Logistic Regression Fitting the Model Assessment and Testing

Logistic Analogs of F-test, R2, etc.

Call: glm(formula = ObamaWin ~ BA, family = "binomial", data = Election08) Deviance Residuals: Min 1Q Median 3Q Max

• 1.7130
• 0.8697

0.2436 0.7314 1.7952 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept)

• 9.4667

2.9656

• 3.192

0.00141 ** BA 0.3706 0.1143 3.243 0.00118 **

• Signif. codes:

0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 69.737

• n 50

degrees of freedom Residual deviance: 49.689

• n 49

degrees of freedom AIC: 53.689 Number of Fisher Scoring iterations: 5

Outline Logistic Regression Fitting the Model Assessment and Testing

“Analysis of Deviance” Likelihood Ratio Test

Instead of an F-statistic, we can compare two models using the (log) likelihood ratio

• Like with R2, in-sample likelihood always goes up

(deviance goes down) if we add a predictor.

• But if it goes up more than expected by chance, that is

evidence the predictor matters.

• 2 × log of the likelihood ratio = Difference in deviance
• Instead of an F-distribution, this statistic has (for large

samples) a χ2 distribution.

Outline Logistic Regression Fitting the Model Assessment and Testing

“Analysis of Deviance”: Likelihood Ratio Test

anova(model, test = "LRT") Analysis of Deviance Table Model: binomial, link: logit Response: ObamaWin Terms added sequentially (first to last) Df Deviance Resid. Df Resid. Dev Pr(>Chi) NULL 50 69.737 BA 1 20.048 49 49.689 7.553e-06 ***

• Signif. codes:

0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1