Testing Specification testing Michel Bierlaire Introduction to - PowerPoint PPT Presentation

Testing Specification testing Michel Bierlaire Introduction to choice models

Differences from classical hypothesis testing

Classical hypothesis testing: example Null hypothesis ( H 0 ) A simple hypothesis contradicting a theoretical assumption. Lady testing tea ◮ Theory: a lady is able to tell if the milk has been poured before of after the tea in a cup. ◮ H 0 : the outcome of the taste is purely random.

Specification testing: example Null hypothesis ( H 0 ) A simple hypothesis contradicting a theoretical assumption. Explanatory variable ◮ Theory: a variable explains the choice behavior. ◮ H 0 : the coefficient of the variable is zero.

Errors in hypothesis testing

Errors in hypothesis testing Type I error

Errors in hypothesis testing Type I error Type II error

Errors in hypothesis testing Type I error Type II error ◮ H 0 rejected and H 0 true.

Errors in hypothesis testing Type I error Type II error ◮ H 0 rejected and H 0 true. ◮ H 0 accepted and H 0 false.

Errors in hypothesis testing Type I error Type II error ◮ H 0 rejected and H 0 true. ◮ H 0 accepted and H 0 false. ◮ Include an irrelevant variable.

Errors in hypothesis testing Type I error Type II error ◮ H 0 rejected and H 0 true. ◮ H 0 accepted and H 0 false. ◮ Include an irrelevant variable. ◮ Omit a relevant variable.

Errors in hypothesis testing Type I error Type II error ◮ H 0 rejected and H 0 true. ◮ H 0 accepted and H 0 false. ◮ Include an irrelevant variable. ◮ Omit a relevant variable. ◮ Loss of efficiency.

Errors in hypothesis testing Type I error Type II error ◮ H 0 rejected and H 0 true. ◮ H 0 accepted and H 0 false. ◮ Include an irrelevant variable. ◮ Omit a relevant variable. ◮ Loss of efficiency. ◮ Specification error.

Errors in hypothesis testing Type I error Type II error ◮ H 0 rejected and H 0 true. ◮ H 0 accepted and H 0 false. ◮ Include an irrelevant variable. ◮ Omit a relevant variable. ◮ Loss of efficiency. ◮ Specification error. ◮ Cost: C I .

Errors in hypothesis testing Type I error Type II error ◮ H 0 rejected and H 0 true. ◮ H 0 accepted and H 0 false. ◮ Include an irrelevant variable. ◮ Omit a relevant variable. ◮ Loss of efficiency. ◮ Specification error. ◮ Cost: C I . ◮ Cost: C II >> C I .

Errors in hypothesis testing Type I error Type II error ◮ H 0 rejected and H 0 true. ◮ H 0 accepted and H 0 false. ◮ Include an irrelevant variable. ◮ Omit a relevant variable. ◮ Loss of efficiency. ◮ Specification error. ◮ Cost: C I . ◮ Cost: C II >> C I . Note In classical hypothesis testing, C I ≈ C II

Impact of an error

Impact of an error Probability of an error P(Type I) =

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true)

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true)

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true) α

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true) α λ

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true) α λ P(Type II) =

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true) α λ P(Type II) = P( H 0 accepted | H 0 false)

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true) α λ P(Type II) = P( H 0 accepted | H 0 false) P( H 0 false)

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true) α λ P(Type II) = P( H 0 accepted | H 0 false) P( H 0 false) β

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true) α λ P(Type II) = P( H 0 accepted | H 0 false) P( H 0 false) β (1 − λ )

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true) α λ P(Type II) = P( H 0 accepted | H 0 false) P( H 0 false) β (1 − λ ) Expected cost Expected cost =

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true) α λ P(Type II) = P( H 0 accepted | H 0 false) P( H 0 false) β (1 − λ ) Expected cost Expected cost = P(Type I) + P(Type II) C I C II

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true) α λ P(Type II) = P( H 0 accepted | H 0 false) P( H 0 false) β (1 − λ ) Expected cost Expected cost = P(Type I) + P(Type II) C I C II = αλ C I + β (1 − λ ) C II

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true) α λ P(Type II) = P( H 0 accepted | H 0 false) P( H 0 false) β (1 − λ ) Expected cost Expected cost = P(Type I) + P(Type II) C I C II = αλ C I + β (1 − λ ) C II Classical hypothesis testing λ ≈ 1, C I ≈ C II

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true) α λ P(Type II) = P( H 0 accepted | H 0 false) P( H 0 false) β (1 − λ ) Expected cost Expected cost = P(Type I) + P(Type II) C I C II = αλ C I + β (1 − λ ) C II Classical hypothesis testing λ ≈ 1, C I ≈ C II : prefer small α .

Impact of an error Probability of an error P(Type I) = P( H 0 rejected | H 0 true) P( H 0 true) α λ P(Type II) = P( H 0 accepted | H 0 false) P( H 0 false) β (1 − λ ) Expected cost Expected cost = P(Type I) + P(Type II) C I C II = αλ C I + β (1 − λ ) C II Specification testing λ ≈ 0 . 5, C II >> C I : larger α can be used.