Linear Regression - Estimating Parameters Bernd Schr oder logo1 - - PowerPoint PPT Presentation

SLIDE 1

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Linear Regression - Estimating Parameters

Bernd Schr¨

• der

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 2

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

General Idea

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 3

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

General Idea

• 1. If we think there is a linear relation y = β0 +β1x between

two variables

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 4

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

General Idea

• 1. If we think there is a linear relation y = β0 +β1x between

two variables, how do we get the coefficients β0 and β1?

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 5

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

General Idea

• 1. If we think there is a linear relation y = β0 +β1x between

two variables, how do we get the coefficients β0 and β1?

• 2. The sum of the squared vertical deviations of the points

(x1,y1),...,(xn,yn) from y = b0 +b1x is f(b0,b1) =

n

∑

i=1

[yi −(b0 +b1xi)]2 .

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 6

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

General Idea

• 1. If we think there is a linear relation y = β0 +β1x between

two variables, how do we get the coefficients β0 and β1?

• 2. The sum of the squared vertical deviations of the points

(x1,y1),...,(xn,yn) from y = b0 +b1x is f(b0,b1) =

n

∑

i=1

[yi −(b0 +b1xi)]2 .

• 3. The linear least squares estimates ˆ

β0 and ˆ β1 of β0 and β1, respectively, are the values that minimize f(b0,b1).

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 7

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

General Idea

• 1. If we think there is a linear relation y = β0 +β1x between

two variables, how do we get the coefficients β0 and β1?

• 2. The sum of the squared vertical deviations of the points

(x1,y1),...,(xn,yn) from y = b0 +b1x is f(b0,b1) =

n

∑

i=1

[yi −(b0 +b1xi)]2 .

• 3. The linear least squares estimates ˆ

β0 and ˆ β1 of β0 and β1, respectively, are the values that minimize f(b0,b1). The estimated regression line or least squares line is the line with y = ˆ β0 + ˆ β1x.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 8

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

General Idea

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 9

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

General Idea

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 10

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

General Idea

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 11

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

General Idea

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 12

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 13

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

∂f ∂b0

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 14

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

∂f ∂b0 = ∂ ∂b0

n

∑

i=1

[yi −(b0 +b1xi)]2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 15

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

∂f ∂b0 = ∂ ∂b0

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

[yi −(b0 +b1xi)]

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 16

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

∂f ∂b0 = ∂ ∂b0

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

[yi −(b0 +b1xi)] ! = 0

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 17

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

∂f ∂b0 = ∂ ∂b0

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

[yi −(b0 +b1xi)] ! = 0 ∂f ∂b1

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 18

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

∂f ∂b0 = ∂ ∂b0

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

[yi −(b0 +b1xi)] ! = 0 ∂f ∂b1 = ∂ ∂b1

n

∑

i=1

[yi −(b0 +b1xi)]2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 19

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

∂f ∂b0 = ∂ ∂b0

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

[yi −(b0 +b1xi)] ! = 0 ∂f ∂b1 = ∂ ∂b1

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

xi [yi −(b0 +b1xi)]

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 20

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

∂f ∂b0 = ∂ ∂b0

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

[yi −(b0 +b1xi)] ! = 0 ∂f ∂b1 = ∂ ∂b1

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

xi [yi −(b0 +b1xi)] ! = 0

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 21

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

∂f ∂b0 = ∂ ∂b0

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

[yi −(b0 +b1xi)] ! = 0 ∂f ∂b1 = ∂ ∂b1

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

xi [yi −(b0 +b1xi)] ! = 0

n

∑

i=1

yi −nb0 −b1

n

∑

i=1

xi =

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 22

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

∂f ∂b0 = ∂ ∂b0

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

[yi −(b0 +b1xi)] ! = 0 ∂f ∂b1 = ∂ ∂b1

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

xi [yi −(b0 +b1xi)] ! = 0

n

∑

i=1

yi −nb0 −b1

n

∑

i=1

xi =

n

∑

i=1

xiyi −b0

n

∑

i=1

xi −b1

n

∑

i=1

x2

i

=

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 23

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

∂f ∂b0 = ∂ ∂b0

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

[yi −(b0 +b1xi)] ! = 0 ∂f ∂b1 = ∂ ∂b1

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

xi [yi −(b0 +b1xi)] ! = 0

n

∑

i=1

yi −nb0 −b1

n

∑

i=1

xi =

n

∑

i=1

xiyi −b0

n

∑

i=1

xi −b1

n

∑

i=1

x2

i

= nb0 +b1

n

∑

i=1

xi =

n

∑

i=1

yi

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 24

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

∂f ∂b0 = ∂ ∂b0

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

[yi −(b0 +b1xi)] ! = 0 ∂f ∂b1 = ∂ ∂b1

n

∑

i=1

[yi −(b0 +b1xi)]2 = −2

n

∑

i=1

xi [yi −(b0 +b1xi)] ! = 0

n

∑

i=1

yi −nb0 −b1

n

∑

i=1

xi =

n

∑

i=1

xiyi −b0

n

∑

i=1

xi −b1

n

∑

i=1

x2

i

= nb0 +b1

n

∑

i=1

xi =

n

∑

i=1

yi b0

n

∑

i=1

xi +b1

n

∑

i=1

x2

i

=

n

∑

i=1

xiyi

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 25

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 26

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

nb0 +b1

n

∑

i=1

xi =

n

∑

i=1

yi b0

n

∑

i=1

xi +b1

n

∑

i=1

x2

i

=

n

∑

i=1

xiyi

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 27

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

nb0 +b1

n

∑

i=1

xi =

n

∑

i=1

yi b0

n

∑

i=1

xi +b1

n

∑

i=1

x2

i

=

n

∑

i=1

xiyi b0 = 1 n

• n

∑

i=1

yi −b1

n

∑

i=1

xi

• Bernd Schr¨
• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 28

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

nb0 +b1

n

∑

i=1

xi =

n

∑

i=1

yi b0

n

∑

i=1

xi +b1

n

∑

i=1

x2

i

=

n

∑

i=1

xiyi b0 = 1 n

• n

∑

i=1

yi −b1

n

∑

i=1

xi

• 1

n

• n

∑

i=1

yi −b1

n

∑

i=1

xi

• n

∑

i=1

xi +b1

n

∑

i=1

x2

i

=

n

∑

i=1

xiyi

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 29

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

nb0 +b1

n

∑

i=1

xi =

n

∑

i=1

yi b0

n

∑

i=1

xi +b1

n

∑

i=1

x2

i

=

n

∑

i=1

xiyi b0 = 1 n

• n

∑

i=1

yi −b1

n

∑

i=1

xi

• 1

n

• n

∑

i=1

yi −b1

n

∑

i=1

xi

• n

∑

i=1

xi +b1

n

∑

i=1

x2

i

=

n

∑

i=1

xiyi b1  

n

∑

i=1

x2

i − 1

n

• n

∑

i=1

xi 2  =

n

∑

i=1

xiyi − 1 n

n

∑

i=1

yi

n

∑

i=1

xi

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 30

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 31

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

b1 = ∑n

i=1 xiyi − 1 n ∑n i=1 yi ∑n i=1 xi

• ∑n

i=1 x2 i − 1 n (∑n i=1 xi)2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 32

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

b1 = ∑n

i=1 xiyi − 1 n ∑n i=1 yi ∑n i=1 xi

• ∑n

i=1 x2 i − 1 n (∑n i=1 xi)2

= ∑n

i=1 xiyi −∑n i=1 yix

• ∑n

i=1 x2 i −∑n i=1 xix

• Bernd Schr¨
• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 33

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

b1 = ∑n

i=1 xiyi − 1 n ∑n i=1 yi ∑n i=1 xi

• ∑n

i=1 x2 i − 1 n (∑n i=1 xi)2

= ∑n

i=1 xiyi −∑n i=1 yix

• ∑n

i=1 x2 i −∑n i=1 xix

• =

∑n

i=1 yi (xi −x)−∑n i=1 y(xi −x)

∑n

i=1 xi (xi −x)−∑n i=1 x(xi −x)

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 34

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

b1 = ∑n

i=1 xiyi − 1 n ∑n i=1 yi ∑n i=1 xi

• ∑n

i=1 x2 i − 1 n (∑n i=1 xi)2

= ∑n

i=1 xiyi −∑n i=1 yix

• ∑n

i=1 x2 i −∑n i=1 xix

• =

∑n

i=1 yi (xi −x)−∑n i=1 y(xi −x)

∑n

i=1 xi (xi −x)−∑n i=1 x(xi −x)

= ∑n

i=1 (xi −x)(yi −y)

∑n

i=1 (xi −x)2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 35

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

b1 = ∑n

i=1 xiyi − 1 n ∑n i=1 yi ∑n i=1 xi

• ∑n

i=1 x2 i − 1 n (∑n i=1 xi)2

= ∑n

i=1 xiyi −∑n i=1 yix

• ∑n

i=1 x2 i −∑n i=1 xix

• =

∑n

i=1 yi (xi −x)−∑n i=1 y(xi −x)

∑n

i=1 xi (xi −x)−∑n i=1 x(xi −x)

= ∑n

i=1 (xi −x)(yi −y)

∑n

i=1 (xi −x)2

b0 = y−b1x

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 36

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Summarizing the Formulas

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 37

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Summarizing the Formulas

1. b1

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 38

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Summarizing the Formulas

1. b1 = ∑n

i=1 xiyi − 1 n ∑n i=1 yi ∑n i=1 xi

• ∑n

i=1 x2 i − 1 n (∑n i=1 xi)2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 39

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Summarizing the Formulas

1. b1 = ∑n

i=1 xiyi − 1 n ∑n i=1 yi ∑n i=1 xi

• ∑n

i=1 x2 i − 1 n (∑n i=1 xi)2

= ∑n

i=1 (xi −x)(yi −y)

∑n

i=1 (xi −x)2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 40

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Summarizing the Formulas

1. b1 = ∑n

i=1 xiyi − 1 n ∑n i=1 yi ∑n i=1 xi

• ∑n

i=1 x2 i − 1 n (∑n i=1 xi)2

= ∑n

i=1 (xi −x)(yi −y)

∑n

i=1 (xi −x)2

= Sxy Sxx

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 41

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Summarizing the Formulas

1. b1 = ∑n

i=1 xiyi − 1 n ∑n i=1 yi ∑n i=1 xi

• ∑n

i=1 x2 i − 1 n (∑n i=1 xi)2

= ∑n

i=1 (xi −x)(yi −y)

∑n

i=1 (xi −x)2

= Sxy Sxx

• 2. b0 = y−b1x

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 42

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Summarizing the Formulas

1. b1 = ∑n

i=1 xiyi − 1 n ∑n i=1 yi ∑n i=1 xi

• ∑n

i=1 x2 i − 1 n (∑n i=1 xi)2

= ∑n

i=1 (xi −x)(yi −y)

∑n

i=1 (xi −x)2

= Sxy Sxx

• 2. b0 = y−b1x
• 3. Fitted (or predicted) values: ˆ

yi := ˆ β0 + ˆ β1xi.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 43

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Summarizing the Formulas

1. b1 = ∑n

i=1 xiyi − 1 n ∑n i=1 yi ∑n i=1 xi

• ∑n

i=1 x2 i − 1 n (∑n i=1 xi)2

= ∑n

i=1 (xi −x)(yi −y)

∑n

i=1 (xi −x)2

= Sxy Sxx

• 2. b0 = y−b1x
• 3. Fitted (or predicted) values: ˆ

yi := ˆ β0 + ˆ β1xi.

• 4. Residuals: yi − ˆ

yi.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 44

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 1

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 45

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 1

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 46

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 1

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 47

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 1

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 48

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 1

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 49

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 1

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 50

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 1

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 51

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 52

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 53

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 54

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 55

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 56

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 57

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 58

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

How Well Does the Line Fit the Data?

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 59

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

How Well Does the Line Fit the Data?

• 1. Error sum of squares (or residual sum of squares).

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 60

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

How Well Does the Line Fit the Data?

• 1. Error sum of squares (or residual sum of squares).

SSE =

n

∑

i=1

(yi − ˆ yi)2 =

n

∑

i=1

• yi −
• ˆ

β0 + ˆ β1xi 2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 61

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

How Well Does the Line Fit the Data?

• 1. Error sum of squares (or residual sum of squares).

SSE =

n

∑

i=1

(yi − ˆ yi)2 =

n

∑

i=1

• yi −
• ˆ

β0 + ˆ β1xi 2 Estimate of σ2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 62

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

How Well Does the Line Fit the Data?

• 1. Error sum of squares (or residual sum of squares).

SSE =

n

∑

i=1

(yi − ˆ yi)2 =

n

∑

i=1

• yi −
• ˆ

β0 + ˆ β1xi 2 Estimate of σ2: ˆ σ2 = s2 = SSE n−2 = ∑n

i=1 (yi − ˆ

yi)2 n−2 .

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 63

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

How Well Does the Line Fit the Data?

• 1. Error sum of squares (or residual sum of squares).

SSE =

n

∑

i=1

(yi − ˆ yi)2 =

n

∑

i=1

• yi −
• ˆ

β0 + ˆ β1xi 2 Estimate of σ2: ˆ σ2 = s2 = SSE n−2 = ∑n

i=1 (yi − ˆ

yi)2 n−2 .

• 2. We divide by n−2 because we have only n−2 degrees of

freedom after determining two parameters ˆ β0 and ˆ β1.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 64

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

How Well Does the Line Fit the Data?

• 1. Error sum of squares (or residual sum of squares).

SSE =

n

∑

i=1

(yi − ˆ yi)2 =

n

∑

i=1

• yi −
• ˆ

β0 + ˆ β1xi 2 Estimate of σ2: ˆ σ2 = s2 = SSE n−2 = ∑n

i=1 (yi − ˆ

yi)2 n−2 .

• 2. We divide by n−2 because we have only n−2 degrees of

freedom after determining two parameters ˆ β0 and ˆ β1. (Previously, using an estimate of µ resulted in the loss of

• ne degree of freedom.)

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 65

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

How Well Does the Line Fit the Data?

• 1. Error sum of squares (or residual sum of squares).

SSE =

n

∑

i=1

(yi − ˆ yi)2 =

n

∑

i=1

• yi −
• ˆ

β0 + ˆ β1xi 2 Estimate of σ2: ˆ σ2 = s2 = SSE n−2 = ∑n

i=1 (yi − ˆ

yi)2 n−2 .

• 2. We divide by n−2 because we have only n−2 degrees of

freedom after determining two parameters ˆ β0 and ˆ β1. (Previously, using an estimate of µ resulted in the loss of

• ne degree of freedom.)
• 3. It can be shown (we cannot present the proof) that S2 is an

unbiased estimator for σ2.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 66

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Formula For SSE

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 67

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Formula For SSE

SSE

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 68

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Formula For SSE

SSE =

n

∑

i=1

(yi − ˆ yi)2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 69

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Formula For SSE

SSE =

n

∑

i=1

(yi − ˆ yi)2 =

n

∑

i=1

• yi −
• ˆ

β0 + ˆ β1xi 2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 70

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Formula For SSE

SSE =

n

∑

i=1

(yi − ˆ yi)2 =

n

∑

i=1

• yi −
• ˆ

β0 + ˆ β1xi 2 =

n

∑

i=1

y2

i −2 ˆ

β0yi −2 ˆ β1xiyi + ˆ β 2

0 +2 ˆ

β0 ˆ β1xi + ˆ β 2

1 x2 i

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 71

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Formula For SSE

SSE =

n

∑

i=1

(yi − ˆ yi)2 =

n

∑

i=1

• yi −
• ˆ

β0 + ˆ β1xi 2 =

n

∑

i=1

y2

i −2 ˆ

β0yi −2 ˆ β1xiyi + ˆ β 2

0 +2 ˆ

β0 ˆ β1xi + ˆ β 2

1 x2 i

=

n

∑

i=1

y2

i −2 ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi +n ˆ β 2

0 +2 ˆ

β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1 n

∑

i=1

x2

i

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 72

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Formula For SSE

SSE =

n

∑

i=1

y2

i −2 ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi +n ˆ β 2

0 +2 ˆ

β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1 n

∑

i=1

x2

i

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 73

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Formula For SSE

SSE =

n

∑

i=1

y2

i −2 ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi +n ˆ β 2

0 +2 ˆ

β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1 n

∑

i=1

x2

i

=

n

∑

i=1

y2

i −2 ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0

• n

∑

i=1

yi − ˆ β1

n

∑

i=1

xi

• Bernd Schr¨
• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 74

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Formula For SSE

SSE =

n

∑

i=1

y2

i −2 ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi +n ˆ β 2

0 +2 ˆ

β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1 n

∑

i=1

x2

i

=

n

∑

i=1

y2

i −2 ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0

• n

∑

i=1

yi − ˆ β1

n

∑

i=1

xi

• +2 ˆ

β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1 n

∑

i=1

x2

i

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 75

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Formula For SSE

SSE =

n

∑

i=1

y2

i −2 ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi +n ˆ β 2

0 +2 ˆ

β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1 n

∑

i=1

x2

i

=

n

∑

i=1

y2

i −2 ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0

• n

∑

i=1

yi − ˆ β1

n

∑

i=1

xi

• +2 ˆ

β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1 n

∑

i=1

x2

i

=

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1 n

∑

i=1

x2

i

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 76

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

SSE =

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1 n

∑

i=1

x2

i

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 77

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

SSE =

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1 n

∑

i=1

x2

i

=

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1

 

n

∑

i=1

x2

i ± 1

n

• n

∑

i=1

xi 2 

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 78

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

SSE =

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1 n

∑

i=1

x2

i

=

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1

 

n

∑

i=1

x2

i ± 1

n

• n

∑

i=1

xi 2  =

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1

 

n

∑

i=1

x2

i − 1

n

• n

∑

i=1

xi 2 + ˆ β 2

1

1 n

• n

∑

i=1

xi 2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 79

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

SSE =

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1

 

n

∑

i=1

x2

i − 1

n

• n

∑

i=1

xi 2 + ˆ β 2

1

1 n

• n

∑

i=1

xi 2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 80

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

SSE =

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi + ˆ β 2

1

 

n

∑

i=1

x2

i − 1

n

• n

∑

i=1

xi 2 + ˆ β 2

1

1 n

• n

∑

i=1

xi 2 =

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi + ˆ β1

• n

∑

i=1

xiyi − 1 n

n

∑

i=1

yi

n

∑

i=1

xi

• + ˆ

β 2

1

1 n

• n

∑

i=1

xi 2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 81

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

SSE =

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi + ˆ β1

• n

∑

i=1

xiyi − 1 n

n

∑

i=1

yi

n

∑

i=1

xi

• + ˆ

β 2

1

1 n

• n

∑

i=1

xi 2

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 82

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

SSE =

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi + ˆ β1

• n

∑

i=1

xiyi − 1 n

n

∑

i=1

yi

n

∑

i=1

xi

• + ˆ

β 2

1

1 n

• n

∑

i=1

xi 2 =

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi − ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi − ˆ β1

n

∑

i=1

xi

• 1

n

n

∑

i=1

yi − ˆ β1 1 n

n

∑

i=1

xi

• Bernd Schr¨
• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 83

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

SSE =

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi −2 ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi + ˆ β1

• n

∑

i=1

xiyi − 1 n

n

∑

i=1

yi

n

∑

i=1

xi

• + ˆ

β 2

1

1 n

• n

∑

i=1

xi 2 =

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi − ˆ β1

n

∑

i=1

xiyi + ˆ β0 ˆ β1

n

∑

i=1

xi − ˆ β1

n

∑

i=1

xi

• 1

n

n

∑

i=1

yi − ˆ β1 1 n

n

∑

i=1

xi

• =

n

∑

i=1

y2

i − ˆ

β0

n

∑

i=1

yi − ˆ β1

n

∑

i=1

xiyi

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 84

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 1.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 85

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 1.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 86

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 1. SSE ≈ 1.1194 and s2 ≈ 0.0622

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 87

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 2.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 88

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 2.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 89

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 2. SSE ≈ 79.7958 and s2 ≈ 4.4331

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 90

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Coefficient of Determination

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 91

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Coefficient of Determination

• 1. Let SST :=

n

∑

i=1

(yi −y)2 be the total sum of squares.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 92

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Coefficient of Determination

• 1. Let SST :=

n

∑

i=1

(yi −y)2 be the total sum of squares. The coefficient of determination is given by r2 = 1− SSE SST .

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 93

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Coefficient of Determination

• 1. Let SST :=

n

∑

i=1

(yi −y)2 be the total sum of squares. The coefficient of determination is given by r2 = 1− SSE SST .

• 2. SSE measures how much variation is not attributed to the

linear relationship.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 94

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Coefficient of Determination

• 1. Let SST :=

n

∑

i=1

(yi −y)2 be the total sum of squares. The coefficient of determination is given by r2 = 1− SSE SST .

• 2. SSE measures how much variation is not attributed to the

linear relationship.

• 3. SST measures the total variation in y.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 95

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Coefficient of Determination

• 1. Let SST :=

n

∑

i=1

(yi −y)2 be the total sum of squares. The coefficient of determination is given by r2 = 1− SSE SST .

• 2. SSE measures how much variation is not attributed to the

linear relationship.

• 3. SST measures the total variation in y.
• 4. The quotient gives the proportion of the variation not

attributed to the linear relationship.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 96

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Coefficient of Determination

• 1. Let SST :=

n

∑

i=1

(yi −y)2 be the total sum of squares. The coefficient of determination is given by r2 = 1− SSE SST .

• 2. SSE measures how much variation is not attributed to the

linear relationship.

• 3. SST measures the total variation in y.
• 4. The quotient gives the proportion of the variation not

attributed to the linear relationship.

• 5. r2 gives the proportion of the variation that is attributed to

the linear relationship.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 97

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 1.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 98

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 1.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 99

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 1. r2 ≈ .9516

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 100

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 2.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 101

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 2.

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters

SLIDE 102

logo1 Coefficients Examples Error Sum of Squares Coefficient of Determination

Example 2. r2 ≈ 0.8403

Bernd Schr¨

• der

Louisiana Tech University, College of Engineering and Science Linear Regression - Estimating Parameters