Regression Pitfalls Pitfall Noun: A hidden or unsuspected danger - - PowerPoint PPT Presentation

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

Regression Pitfalls

Pitfall Noun: A hidden or unsuspected danger or difficulty. A covered pit used as a trap. Multiple regression is a widely used and powerful tool. It is also one of the most abused statistical techniques.

1 / 17 Some Regression Pitfalls Introduction

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

Observational versus Experimental Data

Recall: In some investigations, the independent variables x1, x2, . . . , xk can be controlled; that is, held at desired values. The resulting data are called experimental. In other cases, the independent variables cannot be controlled, and their values are simply observed. The resulting data are called observational.

2 / 17 Some Regression Pitfalls Observational vs Experimental Data

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

Observational example “Cocaine Use During Pregnancy Linked To Development Problems” Two groups of new mothers, 218 used cocaine during pregnancy, 197 did not. IQ tests of infants at age 2 showed lower scores for children of users. “Correlation does not imply causation.”

3 / 17 Some Regression Pitfalls Observational vs Experimental Data

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

The study does not show that cocaine use causes development problems. It does show association, which might be used in prediction. For instance, it could help identify children at high risk of having development problems.

4 / 17 Some Regression Pitfalls Observational vs Experimental Data

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

Experimental example Animal-assisted therapy. 76 heart patients randomly assigned to three therapies: T: visit from a volunteer and a trained dog; V: visit from a volunteer only; C: no visit. Response y is decrease in anxiety.

5 / 17 Some Regression Pitfalls Observational vs Experimental Data

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

Result: ¯ yT = 10.5, ¯ yV = 3.9, ¯ yC = 1.4. Model: E(Y ) = β0 + β1x1 + β2x2, where x1 is the indicator variable for group T and x2 is the indicator variable for group V. The model-utility F-test shows significant differences among groups. Because of random assignment, the differences can be assumed to be caused by the treatments.

6 / 17 Some Regression Pitfalls Observational vs Experimental Data

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

Parameter Estimability

Recall The normal equations X′Xˆ β = X′y that define least squares parameter estimates always have a solution. But if X′X is singular, they have many solutions. An individual parameter that is not uniquely estimated is called nonestimable.

7 / 17 Some Regression Pitfalls Parameter Estimability

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

Example The animal-assisted therapy data. Suppose we tried to fit the model E(Y ) = β0 + β1x1 + β2x2 + β3x3, where x3 is the third indicator variable, for group C. One solution is ˆ β0 = 0, ˆ β1 = ¯ yT = 10.5, ˆ β2 = ¯ yV = 3.9, ˆ β3 = ¯ yC = 1.4. The more usual solution is ˆ β0 = ¯ yC = 1.4, ˆ β1 = ¯ yT − ¯ yC = 9.1, ˆ β2 = ¯ yV − ¯ yC = 2.5, ˆ β3 = 0.

8 / 17 Some Regression Pitfalls Parameter Estimability

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

All estimates change, so no parameter is estimable. The conventional solution is to leave out one variable, or equivalently to constrain one parameter to be zero. Another possibility is to constrain β1 + β2 + β3 = 0, which is appealing in its symmetry, but rarely used in practice. In more complex cases, estimability may be harder to understand.

9 / 17 Some Regression Pitfalls Parameter Estimability

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

Multicollinearity

Two independent variables are orthogonal if their sample correlation coefficient is zero. If all pairs of independent variables are orthogonal, X′X is diagonal, and the normal equations are trivial to solve. In a controlled experiment, the variables are often orthogonal by design. If some pairs are far from orthogonal, the equations may be nearly singular.

10 / 17 Some Regression Pitfalls Multicollinearity

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

If X′X is nearly singular, its inverse (X′X)−1 exists but will have large entries. So the least squares estimates ˆ β = (X′X)−1 X′y are very sensitive to small changes in y. That makes their standard errors large.

11 / 17 Some Regression Pitfalls Multicollinearity

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

Example Carbon monoxide from cigarettes

cigar <- read.table("Text/Exercises&Examples/FTCCIGAR.txt", header = TRUE) pairs(cigar) cor(cigar) summary(lm(CO ~ TAR, cigar)) summary(lm(CO ~ TAR + NICOTINE + WEIGHT, cigar))

The standard error of ˆ βTAR increases nearly five-fold when NICOTINE is added to the model. Note the negative coefficients for NICOTINE and WEIGHT. But both are positively correlated with CO.

12 / 17 Some Regression Pitfalls Multicollinearity

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

Multicollinearity is sometimes measured using the Variance Inflation Factor (VIF). For variable xi, the VIF is VIFi = 1 1 − R2

i

≥ 1, where R2

i is the coefficient of determination in the regression of xi on

the other independent variables {xj, j = i}. VIFi is related to the increase in the standard error of ˆ βi when the

• ther variables are included.

VIFi = 1 if xi is orthogonal to the other independent variables.

13 / 17 Some Regression Pitfalls Multicollinearity

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

Extrapolation

A regression model is an approximation to the complexities of the real world. It may fit the sample data well. If it fits well, it will usually give a reliable prediction for a new context that is similar to those in the sample data. With several variables, deciding when the new context is too different for reliable prediction may be difficult, especially in the presence of multicollinearity.

14 / 17 Some Regression Pitfalls Extrapolation

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

Transformation

In many problems, one or more of the variables (dependent and independent) may be measured and recorded in a form that is not the best from a modeling perspective. Linear transformations are usually pointless, as a linear model is essentially unchanged by it. Among nonlinear transformations, logarithms are most widely useful, followed by powers of the variables.

15 / 17 Some Regression Pitfalls Variable Transformation

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

The primary goal of transformation is to find a good approximation to the way E(Y ) depends on x. Another goal is to make the variance of the random error ǫ = Y − E(Y ) reasonably constant. Finally, if a transformation makes ǫ approximately normally distributed, that is worth achieving.

16 / 17 Some Regression Pitfalls Variable Transformation

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II

Example 7.8 Impact of price of coffee on demand:

coffee <- read.table("Text/Exercises&Examples/COFFEE.txt", header = TRUE) with(coffee, plot(PRICE, DEMAND))

Example 7.8 models Y (DEMAND) against p−1, where p = PRICE. We could also consider log(Y ) and log(p), as well as other powers

• f p.

17 / 17 Some Regression Pitfalls Variable Transformation