STAT 213 Cross-Validation (and Multifactor ANOVA?) Colin Reimer - PowerPoint PPT Presentation

Outline Last Time Cross-Validation STAT 213 Cross-Validation (and Multifactor ANOVA?) Colin Reimer Dawson Oberlin College 12 April 2016

Outline Last Time Cross-Validation Outline Last Time Cross-Validation

Outline Last Time Cross-Validation Reflection Questions How do you decide among all these predictor-selection methods?

Outline Last Time Cross-Validation For Thursday • Read: see last time • Write: finish today’s worksheet • Answer: see last time

Outline Last Time Cross-Validation Outline Last Time Cross-Validation

Outline Last Time Cross-Validation Multicollinearity When one predictor is highly predictable from the other predictors, the model suffers from multicollinearity

Outline Last Time Cross-Validation Multicollinearity When one predictor is highly predictable from the other predictors, the model suffers from multicollinearity One measure: R 2 from a model predicting X j using X 1 , . . . , X j − 1 , X j +1 , . . . , X k .

Outline Last Time Cross-Validation Multicollinearity When one predictor is highly predictable from the other predictors, the model suffers from multicollinearity One measure: R 2 from a model predicting X j using X 1 , . . . , X j − 1 , X j +1 , . . . , X k . Rough rule: If this R 2 is > 0 . 80 , test/intervals for coefficients may not be meaningful.

Outline Last Time Cross-Validation Multicollinearity When one predictor is highly predictable from the other predictors, the model suffers from multicollinearity One measure: R 2 from a model predicting X j using X 1 , . . . , X j − 1 , X j +1 , . . . , X k . Rough rule: If this R 2 is > 0 . 80 , test/intervals for coefficients may not be meaningful. 1 Equivalently: VIF = 1 − R 2 > 5

Outline Last Time Cross-Validation Variance Inflation Factor m.midterm <- lm(Midterm ~ Quiz, data = Scores) summary(m.midterm)$r.squared [1] 0.9498368 m.quiz <- lm(Quiz ~ Midterm, data = Scores) summary(m.quiz)$r.squared [1] 0.9498368 vif(m.both) Midterm Quiz 19.93495 19.93495 vif(m.rotated) V1 V2 1 1

Outline Last Time Cross-Validation Remedies for Multicollinearity 1. Remove redundant predictors 2. Combine predictors into a scale 3. Use the multicollinear model anyway, just don’t use tests/intervals for individual coefficients.

Outline Last Time Cross-Validation Model Selection “Scoring” Adj. R 2 Mallow’s C p Domain Knowledge Best Subset “Search” Forward Selection Backward Selection Stepwise Selection

Outline Last Time Cross-Validation Criteria to "score" models 1. Adj. R 2 : balances fit and complexity for a model in isolation

Outline Last Time Cross-Validation Criteria to "score" models 1. Adj. R 2 : balances fit and complexity for a model in isolation 2. Mallow’s C p / Akaike Information Criterion (AIC): σ 2 estimates mean squared prediction error based on ˆ ε from a “full” model

Outline Last Time Cross-Validation Mallow’s C p / AIC For a model with p coefficients (including the intercept), selected from a pool of predictors, fit using n observations: C p = SSE reduced + 2 p − n (1) MSE full = p + SSE diff (2) MSE full Smaller values correspond to better fit and simpler models.

Outline Last Time Cross-Validation Methods to Explore the Space of Combinations 1. Domain Knowledge: Only build models that make sense 2. Best subset: consider all possible combinations ( 2 k ) 3. Forward selection: start with null model, and consider adding one predictor at a time 4. Backward elimination: start with full model and consider removing one predictor at a time 5. Stepwise regression: consider steps in both directions at each iteration Note: Choose best step based on adj- R 2 or C p /AIC, not based on P -values

Outline Last Time Cross-Validation Outline Last Time Cross-Validation

Outline Last Time Cross-Validation A third dimension What data should we use to (a) Fit the models? (b) Evaluate the models?

Outline Last Time Cross-Validation A third dimension What data should we use to (a) Fit the models? (b) Evaluate the models? Two answers 1. Use all the data for both (what we’ve done so far) 2. Separate the data set into distinct “training” and “validation” sets.

Outline Last Time Cross-Validation In-Sample vs. Out of Sample Prediction • Idea: A good model should make accurate predictions on data it hasn’t seen

Outline Last Time Cross-Validation In-Sample vs. Out of Sample Prediction • Idea: A good model should make accurate predictions on data it hasn’t seen • Evaluating in-sample is subject to overfitting : Since we try to minimize SSE (and maximize SSM), we are liable to extract too much “signal”. Some of the SSM will really be “noise”.

Outline Last Time Cross-Validation In-Sample vs. Out of Sample Prediction • Idea: A good model should make accurate predictions on data it hasn’t seen • Evaluating in-sample is subject to overfitting : Since we try to minimize SSE (and maximize SSM), we are liable to extract too much “signal”. Some of the SSM will really be “noise”. • This is particularly likely if we have lots of model d f .

Outline Last Time Cross-Validation In-Sample vs. Out of Sample Prediction • Idea: A good model should make accurate predictions on data it hasn’t seen • Evaluating in-sample is subject to overfitting : Since we try to minimize SSE (and maximize SSM), we are liable to extract too much “signal”. Some of the SSM will really be “noise”. • This is particularly likely if we have lots of model d f . • Approaches such as adjusted R 2 and Mallow’s C p try to account for overfitting, but why not actually try to predict on different data than used for fitting?

Outline Last Time Cross-Validation Cross-Validation Cross-validation is a technique whereby the full dataset is divided into training and validation (held-out) sets. The first is used for fitting parameters; the second for evaluating predictive power.

Outline Last Time Cross-Validation Cross-Validation Cross-validation is a technique whereby the full dataset is divided into training and validation (held-out) sets. The first is used for fitting parameters; the second for evaluating predictive power. Versions: 1. Two-fold: Divide data (randomly) in half. Fit two models, exchanging roles of training and validation. 2. k -fold: Divide data into k equal sized sets, fit k models letting each set as the validation set. 3. Leave-one-out ( n -fold): Let each observation be its own validation set. Requires fitting n models. Can “score” a model form using average predictive accuracy on