Payo consequences of social learning: A meta analysis Georg - - PowerPoint PPT Presentation

Payo¤ consequences of social learning: A meta analysis

Georg Weizsäcker (LSE) October 2006

I am grateful for funding by the ESRC, grant 1-SEC-T004

Motivation: Why a meta analysis on social learning? "Experiments on social learning are overstudied." Anonymous economist I, 2005 "So, do people learn from others when they should?" Anonymous economist II, 2006 We do not know the answer – when should they? This depends on other players’ unknown strategies. (To generate an answer, we relied on structural models or equi- librium predictions.) ! Use lots of data to estimate the expected payo¤ from learning from others, for a variety of situations.

Motivation II: We lack estimates the payo¤ impact of identi…ed biases The expected payo¤ estimates may also help to quantify the importance of behavioral di¤erences between people/situations: How much more or less would people earn if they behaved di¤erently? Giving excessive weight to own information Not realizing that others have learned Trembling

Higher/lower success in later positions of the game Others? The problem there is, again, that when considering the payo¤ between di¤erent situations/behaviors, we need to control for the value of the available information: The likelihood ratios of the relevant states of the world are di¤erent in di¤erent

• situations. ! Control for the expected payo¤ from the available actions.

Games: All games follow the basic model by Bikhchandani, Hirshleifer and Welch (1992) and were conducted as controlled experiments. The …rst study by Anderson and Holt (1997, symmetric treatment) has the following parameters. t = 0: Nature draws one of two states of the world ("urns"), ! 2 fA; Bg, with Pr(A) = 1=2. Of the balls in urn A, a fraction qA = 2=3 is labelled a and 1 qA is labelled b. Urn B , analogously, has fractions of qB = 2=3 labelled b and 1 qB labelled a. Nature’s draw is not revealed. t = 1: The …rst player makes predictions about the state of the world ("predicts the urn"): The player …rst receives a private signal, in the form of a ball drawn from the true urn. The player then chooses an action d1 2 fA; Bg.

t = 2: The next player observes d1, receives a signal from the same urn, and makes a prediction. t = 3:::T: (Etc. for later players.) All T = 6 players observe the predictions by all previous players, receive a signal, and make a prediction. E.g. for player 6, the information could be AAAAAa, or ABAAAb. If a player’s prediction coincides with the true urn, he or she gets a …xed amount U, here normalized to 1. Otherwise, he or she gets u; normalized to 0.

Data: I re-formatted the raw data from 12 di¤erent studies, all of which had the participants play this game in at least one treatment, with only slight modi…cations. Anderson/Holt (1997, N = 810): 54 participants playing 15 repetitions of the

• game. 18 participants played the symmetric version described above, and 36

participants played an asymmteric treatment with qA = 6=7 and qB = 2=7. Willinger/Ziegelmeyer (1998, N = 324): Replication of Anderson/Holt experi- ment, with q qA = qB = 0:6, 36 participants, and 9 repetitions. Hung/Plott (2001, N = 890). Replication of Anderson/Holt with T = 10 and q = 2=3, in three treatments with minor di¤erences in the experimental

• implementation. 40 participants and 22:25 repetitions on average.

Nöth/Weber (2003, N = 9834). Variant of the game, where q is drawn sep- arately for each player from f0:6; 0:8g, and unknown to other players. 126 participants, and about 78 repetitions on average. Kübler/Weizsäcker (2004, N = 482). Variant where decision makers also decide whether or not they receive a private signal. 36 participants and 15 repetitions, with T = 6 and q = 2=3. Observations are dropped if the participant requested no signal. Drehmann, Oechssler and Roider (2005, N = 2789) 1840 participants played in 8 di¤erent Internet-based treatments with di¤erent signal precisions and di¤erent values for Pr(A). 267 participants were management consultants who played among each other, with T = 7. In addition, 1573 participants of di¤erent

background (mostly students or graduates of universities) played in games with T = 20 (mostly). Participants played up to three repetitions. Ziegelmeyer et al (2002, N = 810). 54 participants and 15 repetitions with T = 9, Pr(A) = 0:55 and q = 2=3. Subjects in one of the two treatments also reported beliefs about the state of the world. Cipriani and Guarino (2003, N = 161). Variant where the players have an

• utside option, modelled after the decision not to trade in …nancial markets. 48

participants and 10 repetitions, with T = 12 and q = 0:7. Observations are dropped of a player or any of his/her predecessor chose the (suboptimal) outside

• ption.

Oberhammer and Stiehler (2003, N = 876). Variant where the subjects also announce their willingness to pay for playing the game, with T = 6, q = 0:6, 36 participants, and about 24 repetitions on average. Alevy, Haigh and List (2005, N = 1647). Replication of both Anderson/Holt treatments, with T = 5 or T = 6, 15 repetitions, and with subjects of di¤erent backgrounds: 55 …nancial market professionals, and 54 undergraduate students. Treatments also di¤ered with respect to gains/losses framing. Goeree et al (2006, N = 8760). 380 participants play four di¤erent long versions

• f the game, with T = 20 and T = 40, q = 5=9 and q = 2=3, and an average
• f 22:7 repetitions.

Dominitz and Hung (2004, N = 2270). 90 participants, games with T = 10 and q = 2=3. 30 participants played 20 repetitions, and the other 60 participants reported belief statements during the last 10 out of 20 repetitions. After dropping some cases from two datasets, the meta dataset contains 29653 individual decision situations fsigi, made by 2795 participants in 34 treatments in 12 separate studies. All of them follow the observation of a private signal and a (possibly empty) string of previous choices made in the analogous situation. In all

• f them there are two actions and two possible payo¤s.

But note the variety in environments, instructions, and paths of play. For each decision situation si, the dataset contains the participant’s choice di as well as the true state of the world !i.

Additional variables for decision si: treatmenti: Two individual decisions (si; sj) made by two participants are in the same treatment if: (i) The participants received the same instructions for the current

• game. (ii) The participants (and their opponents) are drawn from the same pool.

sitcounti: Let Ii be the information available in decision si (observable history within the current game, private signal, e.g. BBAb). sitcounti denotes the number

• f times that a decision situation with the same information Ii occured within the

same treatment. In treatments with A=B symmetry, this includes situations with the symmetric information. (AABa is viewed as identical to BBAb.) The variable sitcount describes how many "identical" situations appear. Let e S(si) be the set of all situations sj that are identical to si, i.e. where the (treatmentj, Ij) description is the same.

This ignores information about history of play before the current run of the game. Under the assumption that history does not enter, the information about others’ signals is identical across e S(si) because all that the participants know about the behavior of other players is re‡ected in (treatment, Ii). Note that sitcount does not compare situations across treatments. Since we know the underlying true state of the world in each case, we can calculate the value of disregarding the own signal. disregi: Indicates that the participant chose not to follow the own signal. est_E[payo¤jdisreg]i: Averaging across situations sj 2 e S(si), what the participant would have earned if he or she had chosen disregi=1. Due to the payo¤ normalization, this is the frequency of the "other" state of the world being true, across sj 2 e S(si). prop_disregi: The proportion of disregi=1, across situations sj 2 e S(si).

.2 .4 .6 .8 1 prop_disreg .2 .4 .6 .8 1 est_E[payoff|disreg] prop_disreg Fitted values

Figure 1: Proportion of disregarding own signal by estimated payo¤. Note: Only situations with sitcounti > 10 are included (339 distinct situations). Regression includes squared and cubed x-variable, weighted by observations.

.2 .4 .6 .8 1 prop_disreg .2 .4 .6 .8 1 est_E[payoff|disreg] prop_disreg Fitted values

Figure 2: Proportion of disregarding own signal by estimated payo¤. Note: Only situations with sitcounti > 10 are included (15592 observations).

We observe that at est_E[payo¤jdisreg]=0, the …tted value of prop_disreg is 0:272 (robust st. err. 0:005), and the regression line reaches 0:5 at est_E[payo¤jdisreg]=0:681. In words, the average decision maker chooses to disregard the signal only if the em- pirical likelihood ratio against the own signal is greater than 2:1. The regression line does not cross through (0:5; 0:5), by a highly signi…cant margin. This con…rms the bias inferred from the structural models of Nöth/Weber (2003), Kübler/Weizsäcker (2004), Goeree et al (2006) and others, that people give more weight to their own signals than to what they could learn from others. Here, any such bias would work on both sides of the graph: If signal con…rms the history, est_E[payo¤jdisreg] is likely to be small, and participants are very likely to not disregard.

The higher variance on the right is not easily explained by adding dummies for position t, or dataset, or strength of the majority. Later positions tend to lie below the regression line often, as do asymmetric games. Note that the same excercise could in principle be done for other choices in experi-

• ments. But the data situation here is expecially good.

Measurement error: Could the result be driven by the imprecision in estimating E[payo¤jdisreg], while the true average line passes through (0:5; 0:5)? With a large measurement error, it may be that many observations with est_E[payo¤jdisreg]> 0:5 correspond to underlying true values of E[payo¤jdisreg]< 0:5, and hence the re- gression line is shifted downwards if there are few observations with a true value of E[payo¤jdisreg]> 0:5. More generally, the estimates are consistent (for sitcount! 1) but not unbiased. Three comforting obserations: The performance in situations with est_E[payo¤jdisreg]< 0:5 is very good. If measurement error is symmetric, we would expect some observations scattered in the left half of the graph.

We can estimate the standard deviation of the estimated variable est_E[payo¤jdisreg], which measures the size of the measurement error. Across situations si the mean of this standard deviation is 0:0527, far below the estimated deviation from (0:5; 0:5). Increasing the cuto¤ to sitcount>30 or sitcount>50 does not change the regres- sion line much.

10 20 30 40 Density .05 .1 .15 std_est_E[payoff|disreg]

Figure 3: Distribution of standard deviations of expected payo¤ estimates. Note: Sample of observations is restricted to cases with sitcount>10.

.2 .4 .6 .8 1 prop_disreg .2 .4 .6 .8 1 est_E[payoff|disreg] prop_disreg Fitted values

Figure 4: Proportion of disregarding own signal by estimated payo¤. Note: Only situations with sitcounti > 10 are included (339 distinct situations). Regression includes squared and cubed x-variable, weighted by observations.

.2 .4 .6 .8 1 prop_disreg .2 .4 .6 .8 1 est_E[payoff|disreg] prop_disreg Fitted values

Figure 5: Proportion of disregarding own signal by estimated payo¤. Note: Only situations with sitcounti > 30 are included (124 distinct situations). Regression includes squared and cubed x-variable, weighted by observations. Line passes through (0:5; 0:253) and (0:688; 0:5).

.2 .4 .6 .8 1 prop_disreg .2 .4 .6 .8 est_E[payoff|disreg] prop_disreg Fitted values

Figure 6: Proportion of disregarding own signal by estimated payo¤. Note: Only situations with sitcounti > 50 are included (68 distinct situations). Regression in- cludes squared and cubed x-variable, weighted by observations. Line passes through (0:5; 0:250) and (0:662; 0:5).

Trade-o¤ between variety and precision of generated regressors: For almost any subsample, we have enough observations, but for the meta analysis, we want a large number of distinct situations, so that the results are not driven by speci…c subsamples and so that we can compare across interesting dimensions. But we also want high values of sitcount, to estimate the payo¤ from the available actions accurately. ! Restrict the data to those above a lower bound of sitcount, but only enough to preserve some variety. Restricting the sample to sitcount>10: Observations split up by player position t:

# of

• bs.

mean (sitcount) from largest data set history of only A

• r only B

t=1

3135 463 52.0% (NW)

• t=2

2980 267 55.0% (NW) 100%

t=3

2699 143 60.7% (NW) 63.2%

t=4

2367 87 69.2% (NW) 49.1%

t=5

2012 64 81.5% (NW) 42.1%

t=6

1844 52 83.7% (NW) 39.9%

t=7

136 36 83.1% (DH) 100.0%

t=8

134 34 82.1% (DH) 100.0%

t=9

139 29 91.4% (DH) 91.4%

t=10

128 33 82.0% (DH) 100.0%

t>10

• total/ave.

15592 198 (ave.) 62.5% (NW) 64.1%

The variety with respect to position t and frequencies of only-A=B histories seems …ne. Also, sitcount is high on average. But dataset by Nöth/Weber (2003) is rather dominant, because there all the data were collected in a single treatment, so that sitcount is high for a large number of observations: 9738 out of 9834 satisfy the requirement that sitcount> 10 in this dataset. But in the number of distinct situa- tions, this dataset is less dominant. And we can always run robustness checks, where NW is left out of the sample. Restricting the sample to sitcount>10: Distinct situations split up by player position t:

# of distinct situations mean (sitcount) from largest data set history of only A

• r only B

t=1

37 85 32.4% (AHL)

• t=2

50 60 20.0% (AHL) 100.0%

t=3

57 47 26.3% (GMPR) 64.9%

t=4

58 41 27.6% (NW) 53.4%

t=5

52 39 61.5% (NW) 42.3%

t=6

64 29 76.6% (NW) 28.1%

t=7

5 27 60.0% (DH) 100.0%

t=8

5 27 60.0% (DH) 100.0%

t=9

6 23 83.3% (DH) 83.3%

t=10

5 26 60.0% (DH) 100.0%

t>10 total/average

339 46 42.3% (NW) 58.9%

Estimating the payo¤ di¤erences that are due to ’behavioral biases’: Several papers – and the evidence above – suggest that people give excessive weight to their own signal. How much do people give up due to this e¤ect? Here, we have no structural model, hence no benchmark. But we can hypothesize that the ’bias’ has a negative e¤ect in particular situations. !Calculate the payo¤ di¤erence between these situations and others. For example, consider the following dummy variable: countersigi: Indicates whether the participant’s own signal runs counter to a strict

• majority. E.g. AAABb or Ab.

The above discussion suggests that people loose money when countersig=1:

How much more do people earn when countersig=0 than when countersig=1? (Note the selection problem.) How much wold they earn in each case if they always choose the better action? How much more would the participants earn in situations where countersig=1 if they were as successful in identifying the optimal action as they are in the other situations? How much less would the participants earn in situations where countersig=0 if they were as successful in identifying the optimal action as they are in situations with countersig=1?

Decomposition of payo¤ into own versus others’ fault: Fix a situation si. Let p(ak

i jxi) be the probability that a participant chooses ak i , where k 2 f1; 2g

and the variables xi = x1

i ...xm i

are situation-speci…c variables, such as treatmenti, est_E[payo¤jdisreg]i, etc. The participant’s expected payo¤ E[ijxi] is given by E[ijxi] = p(a1

i jxi) E[ija1 i ; xi] + p(a2 i jxi) E[ija2 i ; xi]

= p(a1

i jxi) E[ija1 i ; xi] + (1 p(a1 i jxi)) (1 E[ija1 i ; xi])

= p(a1

i jxi) (2E[ija1 i ; xi] 1) + 1 E[ija1 i ; xi]

The participant’s realized payo¤ is subject to randomness in the urn and signal draws, as well as in the behavior of others. But as before, we can estimate E[ija1

i ; xi] from

the data without relying on a structural model.

To address the above behavioral-counterfactual question, vary p(a1

i jxi), and calculate

the e¤ect on E[ijxi]. The counterfactual is just for one participant, i.e. E[ija1

i ; xi]

is viewed as constant.

E[ijxi] = p(a1

i jxi) (2E[ija1 i ; xi] 1) + 1 E[ija1 i ; xi]

Specifying a1

i and p(a1 i jxi): We are free to choose a labelling of a1 i . For conve-

nience, let a

i be the action with the higher estimated value E[ijak i ; xi], k 2 f1; 2g.

Hence, a

i is an estimate of the better action, and E[ija i ; xi] 0:5 whenever the

estimate is correct. As ’behavioral assumption’ specify the form of p(a

i jxi). Then we can create coun-

terfactuals through shifts in the variables xi. For example, what is the e¤ect on E[ijxi] if countersig changes from 0 to 1, controlling for the value of the avail- able information. (Note that we do not impose (yet) that the participants know this value.) This avoids the selection problem that in situations with countersig=1, the likelihood ratio may be di¤erent from situations with countersig=0 – due to both the information structure and others’ behavior.

Summary: A three-step procedure:

• 1. Estimate E[ija

i ; xi] using the frequencies of the underlying urns.

• 2. Estimate coe¢cients of p(a

i jxi) using the choices, generate p(a i jxi) of interest.

• 3. Calculate E[ijxi] = p(a

i jxi)(2E[ija i ; xi]1) from the above estimates.

Alternative speci…cations of p(a

i jxi):

p(a

i ) = + countersigi

• Coe¢cient is di¤erence between averages of

choosing a

i .

p(a

i ) = + countersigi + g(E[ija i ; xi])

• controlling for payo¤ di¤er-

ence between the two actions. p(a

i ) = + countersigi + c0Ii

• adding dummies for position t,

treatment, and strength of majority p(a

i ) = + countersigi + g(E[ija i ; xi]) + c0Ii

• both

.2 .4 .6 .8 1 prop_a* .5 .6 .7 .8 .9 1 est_E[payoff|a*]

Figure 7: Proportion of participants using the estimated optimal action a. Note: Only observation with sitcount> 10 included.

.2 .4 .6 .8 1 prop_a* .5 .6 .7 .8 .9 1 est_E[payoff|a*] propopt propopt Fitted values Fitted values

Figure 8: Proportion of participants using the estimated optimal action a, separated by countersig (1=red). Note: Only observation with sitcount> 10 included. Regres- sion includes simple, squared and cubed term for est_E[payo¤ja], and dummy for countersig.

p(ajx)

(1) (2) (3) (4) countersig

• 0.323 (.008)
• 0.202 (.008)
• 0.366 (.008)
• 0.222 (.009)

est_E[payo¤ja] no yes no yes

• ther controls

no no yes no # of obs. 15592 15592 15592 15592

R2

0.126 0.248 0.193 0.269 realized earningsj1 0.550 0.550 0.550 0.550 realized earningsj0 0.718 0.718 0.718 0.718 maximum earningsj1 0.648 (...) 0.648 (...) 0.648 (...) 0.648 (...) maximum earningsj0 0.748 (...) 0.748 (...) 0.748 (...) 0.748 (...)

earningsj1 ! 0

0.096 (.002) 0.060 (.002) 0.109 (.002) 0.066 (.003)

earningsj0 ! 1

0.160 (.004)

• 0.100 (.004)
• 0.181 (.004)
• 0.110 (.005)

Column (2) of the table shows that if the participants were as successful in couter- sig=1 situations as they are in countersig=0 situations – controlling for the value of the available information, then they would increase their earnings from 0:55 to about 0:61. Hence, the increase of earnings over randomization (which yields 0:5) would be more than doubled. Reversely, if the participants in countersig=0 situations would behave as unsuccess- fully as they do in countersig=1 situations, their earnings increase over randomization would decrease by almost half. Either way, large proportion of the payo¤ di¤erence can be attributed to a di¤erence in behavior between the two classes of situations.

Other dummy variable of interest: position_gt4i: position t 4.

p(ajx)

(1) (2) (3) (4) typegt4 0.053 (.007)

• 0.047 (.007)

0.077 (.009)

• 0.032 (.008)

est_E[payo¤ja] no yes no yes

• ther controls

no no yes no # of obs. 15592 15592 15592 15592

R2

0.004 0.208 0.056 0.232 realized earningsj1 0.699 0.699 0.699 0.699 realized earningsj0 0.635 0.635 0.635 0.635 maximum earningsj1 0.766 (...) 0.766 (...) 0.766 (...) 0.766 (...) maximum earningsj0 0.676 (...) 0.676 (...) 0.676 (...) 0.676 (...)

earningsj1 ! 0

• 0.028 (.004)

0.025 (.004)

• .041 (.005)

0.017 (.004)

earningsj0 ! 1

0.019 (.002)

• 0.016 (.002)

0.027 (.003)

• 0.011 (.003)

• thers_disregardedi: A dummy indicating whether it was relatively likely that the

average predecessor in the game has disregarded his or her signal. E.g. for the case

• f a player 4 whose infomration is AAAb: (i) Calculate for all three predecessors

the ex-post likelihood (knowing all datafrom this treatment) that their decision was generated by someone who did not follow the signal. This is done e.g. by counting how many AAA decisions come from AAb in this treatment. (ii) Take average over the three predecessors. (iii) Indicate whether the average is greater than the median

• f such averages in the dataset.

The result on this variable are fairly weak and unreliable, until now. disregopti: Indicates that est_E[payo¤jdisreg]i 0:5.

.2 .4 .6 .8 1 prop_disreg .2 .4 .6 .8 1 est_E[payoff|disreg] prop_disreg Fitted values

Figure 9: Proportion of disregarding own signal by estimated payo¤. Note: Only situations with sitcounti > 10 are included (339 distinct situations). Regression includes squared and cubed x-variable, weighted by observations.

The analogous calculations show that when disregopt= 1, the participants earn 0:528

• ut of an estimated maximum earnings level of 0:640 – barely more than from ran-

domization, which would yield 0:5. In situations with disregopt=0, they earn 0:726

• ut of an estimated maximum level of 0:750.

Conclusions: When people should learn from others, they behave di¤erently, but are not much more successful than pure randomization. Until now, the preference for following the own signal seems much stronger than

• ther e¤ects.

Method may be useful for other dataset. Some estimates of variance rely on large numbers, but most of the means do not.