Testing the rationality of categorical predictions Carlos Madeira - - PowerPoint PPT Presentation

Testing the rationality of categorical predictions

Carlos Madeira

Central Bank of Chile

May, 2017

Carlos Madeira (Central Bank of Chile) Rationality of categorical predictions May, 2017 1 / 18

Introduction

Data on subjective expectations is now common, especially in education studies (Dominitz and Manski, 1996, Manski, 2004, Zafar, 2011, Attanasio and Kaufmann, 2014, Giustinelli, 2016). In the last 20 years studies have used more accurate measures of subjective expectations (Delavande and Rohwedder, 2008), including its central tendency (median, mode, mean) and uncertainty (Std, IQR, IDR). However, many datasets still use less formal or less precise measures of beliefs, including point predictions and qualitative statements (Manski, 2004). Some surveys are already quite long and exhaustive, but need some info on beliefs that may affect future actions/investments - see Michigan Survey of Consumers (MSC) or the ECB’s Household Finance and Consumption Survey (HFCS). In asset portfolio choice, studies show that retail investors are unable to use standard-errors and other uncertainty measures, but can rank risk choices (Hackethal and Inderst, 2011).

Carlos Madeira () Rationality of categorical predictions May, 2017 2 / 18

This work

I build a test of rationality for categorical predictions under the assumption that agents provide their subjective mode or a fixed-quantile. Manski (1990) already provides a test of rationality for binary outcomes using a moment inequality and assuming agents report the median (=mode) outcome. Das, Dominitz and Soest (1999) extend Manski’s analysis to the general case of multiple ordered categorical expectations, under a subjective mode or a fixed-quantile, showing it requires testing several moment inequalities. DDS (1999) do not provide a statistical test of whether these inequalities jointly hold. I use a multiple moment inequality statistic which is asymptotically distributed as a weighted chi-square distribution (Kudo, 1963, Wolak, 1987, Rosen 2008) when all the inequalities bind and rejects the null-hypothesis with a confidence level equal or smaller than θ.

Carlos Madeira () Rationality of categorical predictions May, 2017 3 / 18

First: Test against a prediction rule using only observables

Let Yi be the actual outcome, PR

i = g(Xi, εp i ) the respondents’ guess and

Pi = f (Xi) an alternative prediction rule f (.). Outcomes Y are ordered: 1, ..., K. Xi is observable and εp

i is private information. If respondents use

the loss function L(.) then rationality demands: 1.1) H0: µ ≡ E[L(Yi, Pi = f (Xi)) − L(Yi, PR

i ) | Xi] ≥ 0.

The null hypothesis of rationality H0: µ = 0 vs. H1: µ < 0 is tested by: 1.2) t = √n 1

N ∑N i=1 L(Yi, Pi = f (Xi)) − L(Yi, PR i )

ˆ Std(L(Yi, Pi = f (Xi)) − L(Yi, PR

i ))

• → N(0, 1),

Valid under three assumptions. A.1) Yi, PR

i

are independent across i (which excludes aggregate shocks); A.2) the loss function L(.) is known both by the agents and the econometrician; A.3) the test is only valid against a specific prediction rule given by f (Xi). In a sense this simple test treats the categorical outcomes as numeric values.

Carlos Madeira () Rationality of categorical predictions May, 2017 4 / 18

Test of a fixed-quantile rationality hypothesis

A.1) plus less restrictive version of A.2), which is that outcomes are

• rdered. α-quantile rationality implies:

2.1) c1 > 0, with c1 ≡ α − Pr(Yi ≤ k − 1 | Xi, PR

i = k),

2.2) c2 ≥ 0, with c2 ≡ (1 − α) − Pr(Yi ≥ k + 1 | Xi, PR

i = k).

Let Pj|k = Pr(Yi = j | Xi, PR

i = k) and its estimate

ˆ Pj|k = ∑N

i=1 1(Yi = j, xi, PR i = k)

nk , where nk = ∑N

i=1 1(xi, PR i = k).

Statistics for testing c1 > 0 and c2 ≥ 0 separately are: 2.3) t1(α−quantile) = √nk(α − ∑k−1

j=1 ˆ

Pj|k − c1), 2.4) t2(α−quantile) = √nk((1 − α) − ∑K

j=k+1 ˆ

Pj|k − c2). t = [t1, t2] ∼ N(0, Σ∗). A valid asymptotic test of H0 : c1 > 0, c2 ≥ 0 is provided by W = infc[t( ˆ Σ∗)−1t st c ≥ 0], with H0 rejected if W > ¯ χθ

1.

Carlos Madeira () Rationality of categorical predictions May, 2017 5 / 18

Test of a subjetive mode rationality

The mode can be applied to purely qualitative outcomes. 3.1) λj|k ≥ 0, with λj|k ≡ Pr(Yi = k | xi, PR

i = k) − Pr(Yi = j | xi, PR i = k), ∀j = k,

H0) λ(k) = [λ1|k, .., λK |k] ≥ 0 can be tested using: 3.2) t(j | k) =√nk( ˆ Pk|k − ˆ Pj|k − λj|k), j = k. t(k) ≡ [t(1 | k), .., t(K | k)] ∼ N(0, Σ∗∗). A valid asymptotic test of H0 : λ(k) ≥ 0 is provided by W (k) = infλ(k)[t(k)( ˆ Σ∗∗)−1t(k) st λ(k) ≥ 0], with H0 rejected if W (k) > ¯ χθ

K −1.

Carlos Madeira () Rationality of categorical predictions May, 2017 6 / 18

Multiple inequalities imply non-normal asymptotics

¯ χθ

K −1 is a critical value of a weighted average of chi-square distributions of

degree 0 to K − 1, with weights w(K − 1, h, Σ∗∗) in Wolak (1987): Pr( ¯ χK −1 > ¯ χθ

K −1) = ∑K −1 h=0 w(K − 1, h, Σ∗∗) Pr(χ2 h > ¯

χθ

K −1).

The asymptotic Type I error of this test is equal to or smaller than θ. W (k) is asymptotically distributed as ¯ χK −1 only when H0 is valid and all the inequalities are binding. If H0 is valid and only m inequalities of the vector λ(k) bind (with m < K − 1), then W (k) is asymptotically distributed as ¯ χm and the test has a Type I error smaller than θ (Rosen, 2008). If H0 is valid and no inequalities bind, then W (k) goes asymptotically to 0.

Carlos Madeira () Rationality of categorical predictions May, 2017 7 / 18

Monte Carlo Study: Scenarios

Scenarios to test whether the Mode or Median is rejected as the probability of each outcome given a forecast, Pr(Yi = j | PR

i = k)

Scenario PR

i = k /Yi = j

j =1 2 3 4 H0 valid? Yes/No: Binding constraints Mode Median 1 k = 1 0.50 0.35 0.10 0.05 Yes: 0 Yes: 1 2 2 0.25 0.25 0.25 0.25 Yes: 3 Yes: 1 3 3 0.05 0.20 0.25 0.50 No: 0 Yes: 1 4 4 0.05 0.15 0.40 0.40 Yes: 1 No: 0 Others: j = 1 2 3 4 5 k = 2 0.15 0.35 0.35 0.15 Yes: 1 Yes: 1 6 2 0.30 0.35 0.25 0.10 Yes: 0 Yes: 0 7 3 0.10 0.35 0.30 0.25 No: 0 Yes: 0

Carlos Madeira () Rationality of categorical predictions May, 2017 8 / 18

Type I error Mode: 10,000 MC samples

Type I error Mode (%): θ ≤5% Multi-condition test Das-Dominitz-Soest test Scenario N=25 50 100 250 N=25 50 100 250 1 0.3% 0.1% 0.0% 0.0% 1.3% 0.4% 0.1% 0.0% 2 7.7% 6.1% 5.5% 5.1% 19.6% 17.9% 15.6% 15.0% 3 NA NA 4 2.7% 2.2% 2.2% 2.2% 8.2% 7.0% 6.7% 6.2% 5 2.7% 2.0% 1.8% 2.1% 9.2% 7.4% 6.9% 6.1% 6 1.9% 1.0% 0.5% 0.1% 6.0% 3.7% 1.9% 0.7% 7 NA NA

Carlos Madeira () Rationality of categorical predictions May, 2017 9 / 18

Type I error Median: 10,000 MC samples

Type I error Median (%): θ ≤5% Multi-condition test Scenario N=25 50 100 250 500 1 5.5% 6.0% 4.2% 5.6% 5.4% 2 5.8% 5.8% 4.5% 5.8% 4.8% 3 5.2% 6.0% 4.7% 6.1% 5.1% 4 NA 5 5.5% 5.8% 4.5% 5.7% 4.8% 6 0.1% 0.0% 0.0% 0.0% 0.0% 7 1.8% 1.1% 0.5% 0.1% 0.0% Multi-condition is only slightly better than the DDS test and only in samples lower than 25 observations.

Carlos Madeira () Rationality of categorical predictions May, 2017 10 / 18

Application: The Beginnning School Study (BSS)

Extensive panel data on expectations from the BSS (1982-2002). 838 families selected with children in first-grade in 1982 from 20 Baltimore Public Schools. On average 80% of the sample is observed in each year. Stratified selection (no expansion factors/survey weights given). Poor socioeconomic background of the BSS families. Expectations and outcomes of academic scores. Others: time studying, education and occupation as adults. "Aspirations" and "Best Guesses" are separate. Outcomes are reported directly from administrative sources (schools). Parents interviewed at home or by phone. Children in school. Parent (Fall) and Student (Fall, Spring) interviews done at the beginning

• f the academic quarter. Teacher interviews at the end of the year.

Carlos Madeira () Rationality of categorical predictions May, 2017 11 / 18

Maternal age and education

Table 2.1: Age and education level of students’ parents Age at birth mother Education mother 10-19 28.1% Grade 9 or Less 18.9% 20-25 40.4% Some High-school 19.7% 26-30 20.7% Graduate High-school/ GED 34.2% 31-35 8.1% Some College 17.0% 36-52 2.7% Finish 4-year College 6.1% Missing 10.5% Advanced Degree 4.1% Missing 6.6%

Carlos Madeira () Rationality of categorical predictions May, 2017 12 / 18

Math marks and guesses

Guesses and Marks Best guess for Math (Fall) Actual Marks Parents Students Fall Letter Mark 82 82-90 82 82-90 82 82-90 1 - Unsatisfactory 3.2% 2.5% 2.6% 2.7% 17.7% 20.6% 2 - Satisfactory 35.6% 30.3% 12.5% 18.9% 49.4% 46.0% 3 - Good 48.7% 47.1% 37.4% 44.9% 24.7% 26.2% 4 - Excellent 12.5% 20.1% 47.6% 33.6% 8.2% 7.3% Respondents vs Median % of correct guesses Absolute Midpoint Loss Q1 Math English Math English Median of past marks 44.3 52.5 7.6 6.4 Parental guess 32.7∗∗∗ 33.2∗∗∗ 10.2∗∗∗ 9.6∗∗∗ Student guess 29.4∗∗∗ 23.6∗∗∗ 11.1∗∗∗ 12.5∗∗∗ Teacher’s guess 44.9 50.0∗∗∗ 7.4 6.5 Math Teacher’s guess 38.1 40.3 9.4∗∗∗ 8.4 English Teacher’s guess 43.9 42.6 8.2 8.4∗∗

Carlos Madeira () Rationality of categorical predictions May, 2017 13 / 18

Multi-condition median tests: first-grade

Groups of Respondents Median p-value % Mode p-value % Guess Guess Students (Math Q1) U S G E U G E Less than HS: 1982 23.3 97.6 0.0 0.0 49.5 0.0 0.0 High School (HS): 1982 25.9 97.5 3.8 0.0 50.5 23.0 0.0 More than HS: 1982 0.0 91.8 32.5 0.0 0.0 63.9 0.0 All guesses* 54.1 97.6 0.0 0.0 62.9 0.0 0.0 Parents (Math Q1) U S G E U G E Less than HS: 1982 91.5 91.7 0.0 0.0 57.6 0.0 2.1 High School (HS): 1982 50.0 93.4 1.7 0.0 57.6 1.9 0.7 More than HS: 1982 0.0 92.8 80.3 0.0 44.7 64.3 3.1 All guesses* 88.0 95.2 0.0 0.0 61.9 0.0 0.0

Carlos Madeira () Rationality of categorical predictions May, 2017 14 / 18

Multi-condition mode tests (conditional on past median)

Students (Math Q1) Median mark: U S G E Guess: U 63.6 61.5 50.5 S 3.7∗∗ 62.8 66.6 63.6 G 0.0∗∗∗ 0.0∗∗∗ 67.7 65.2 E 0.0∗∗ 0.0∗∗∗ 0.0∗∗∗ 65.0 Parents (Math Q1) Median mark: U S G E Guess: U 57.6 64.7 57.6 S 61.2 65.1 67.6 57.6 G 0.0∗∗∗ 0.0∗∗∗ 11.8 65.0 E 0.0∗∗∗ 0.0∗∗∗ 0.0∗∗∗ 65.1 Statistical significance at the level of ∗∗∗ 1%, ∗∗ 5%, ∗ 10%.

Carlos Madeira () Rationality of categorical predictions May, 2017 15 / 18

Impact of prediction ability on time of study

Regressions (OLS) of Time Fall (1987-1990) Spring (1986-94) spent studying in Math Model 1 Model 2 Model 1 Model 2 Parents’ Math prediction 0.633 1.472** success (0.973) (0.714) Students’ Math prediction 2.962** 0.157 success (1.164) (0.766) Parents’ mean mid-point

• 0.0528
• 0.0894**

loss of past guesses (0.0559) (0.0399) Students’ mean mid-point

• 0.0926*

0.00675 loss of past guesses (0.0539) (0.0395) Median of Past Math marks 0.123*** 0.108** 0.115*** 0.110*** (midpoint value) (0.0412) (0.0452) (0.0311) (0.0332) Other Controls: Years Nr of observations 1,793 1,793 3,926 3,926

Carlos Madeira () Rationality of categorical predictions May, 2017 16 / 18

Evolution of parents/students predictions

.2 .3 .4 .5 .6 .7 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 Year Median Q1 Median Q4 Parents Q1 Students Q1 Students Q4

Math

.2 .3 .4 .5 .6 .7 1982 1983 1984 1985 1986 1987 1988 1989 1990 Year Median Q1 Median Q4 Parents Q1 Students Q1 Students Q4

English/Reading

Carlos Madeira () Rationality of categorical predictions May, 2017 17 / 18

Conclusions

I build a test of the necessary conditions for Median or Mode rationality in forecasts of qualitative outcomes. Bad: asymptotic distribution is ugly. Monte Carlo: negliglible difference for the Median, but a significant improvement for the Median. Application for Baltimore families: Parents and students that forecast performance over the past are rejected to be rational (except parents with U that forecast S). Prediction success in the past tends to have a positive impact on time spent studying Math, but not English.

Carlos Madeira () Rationality of categorical predictions May, 2017 18 / 18