Hypothesis Testing Part III: Pretest Bias James J. Heckman - PowerPoint PPT Presentation

Hypothesis Testing Part III: Pretest Bias James J. Heckman University of Chicago Based on T.A. Bancroft, The Annals of Mathematical Statistics , Vol. 15, No. 2. (June, 1944), pp. 190-204. Econ 312 Spring 2019 1

� � � � � � � A common procedure almost universal in economics is to select a model on the basis of a preliminary test. Example: = � � � + � � � ( � � | � � ) = 0 = [ � 1 � � � 2 � ] = ( � 1 � � 2 ) Include � 2 if a “ � ” statistic on ˆ � 2 is big enough. 2

ˆ ˆ ˜ � 1 ,pretest = � 1 1 ( � � 2 � � ) + � 1 1 ( � � 2 � � ) ˆ � 1 comes from unrestricted model; ˜ � 1 comes from model deleting � 2 . 3

For a fixed size test, say � , � % of time we reject when null � 0 : � 2 = 0 is true. � Estimator is inconsistent unless we change � with sample size (e.g. � � � 0 ). Bayesians show it’s inadmissible under quadratic loss. Examples of Bias: 4

Model : Y i = X i β +  i ; i = 1,...,N ρ X 1 0, 1 X ≡ ∼ N ; iid ρ X 2 1  ∼ N  0, σ 2  ; iid Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.9 σ =10 β 2 , ρ =0.9 σ =10 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.9; σ 2 = 10 N = 50, ρ = 0.9; σ 2 = 10 5

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.75 σ =10 β 2 , ρ =0.75 σ =10 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.75; σ 2 = 10 N = 50, ρ = 0.75; σ 2 = 10 6

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.5 σ =10 β 2 , ρ =0.5 σ =10 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.5; σ 2 = 10 N = 50, ρ = 0.5; σ 2 = 10 7

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.25 σ =10 β 2 , ρ =0.25 σ =10 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.25; σ 2 = 10 N = 50, ρ = 0.25; σ 2 = 10 8

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.9 σ =5 β 2 , ρ =0.9 σ =5 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.9; σ 2 = 5 N = 50, ρ = 0.9; σ 2 = 5 9

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 1 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 1 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.75 σ =5 β 1 , ρ =0.75 σ =5 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.75; σ 2 = 5 N = 50, ρ = 0.75; σ 2 = 5 10

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.5 σ =5 β 2 , ρ =0.5 σ =5 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.5; σ 2 = 5 N = 50, ρ = 0.5; σ 2 = 5 11

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.25 σ =5 β 2 , ρ =0.25 σ =5 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.25; σ 2 = 5 N = 50, ρ = 0.25; σ 2 = 5 12

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.9 σ =2 β 2 , ρ =0.9 σ =2 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.9; σ 2 = 2 N = 50, ρ = 0.9; σ 2 = 2 13

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.75 σ =2 β 2 , ρ =0.75 σ =2 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.75; σ 2 = 2 N = 50, ρ = 0.75; σ 2 = 2 14

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 2 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 2 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 2 , ρ =0.5 σ =2 β 2 , ρ =0.5 σ =2 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.5; σ 2 = 2 N = 50, ρ = 0.5; σ 2 = 2 15

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.25 σ =2 β 2 , ρ =0.25 σ =2 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.25; σ 2 = 2 N = 50, ρ = 0.25; σ 2 = 2 16

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.9 σ =1 β 2 , ρ =0.9 σ =1 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.9; σ 2 = 1 N = 50, ρ = 0.9; σ 2 = 1 17

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.75 σ =1 β 2 , ρ =0.75 σ =1 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.75; σ 2 = 1 N = 50, ρ = 0.75; σ 2 = 1 18

Pre-Test Estimator Density Function Pre-Test Estimator Density Function β 1 density without drop of β 2 β 2 density without drop of β 2 β 1 N = 50 β 1 N = 50 β 1 density with drop of β 2 β 2 density with drop of β 2 1.2 1.2 Probability Density Function of estimated Probability Density Function of estimated 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 -2 -1 0 1 2 3 4 -2 -1 0 1 2 3 4 β 1 , ρ =0.5 σ =1 β 2 , ρ =0.5 σ =1 Distribution of β 1 Distribution of β 2 N = 50, ρ = 0.5; σ 2 = 1 N = 50, ρ = 0.5; σ 2 = 1 19