Robustness and idealizations in agent-based models of scientific - - PowerPoint PPT Presentation

Robustness and idealizations in agent-based models of scientific interaction

Daniel Frey1 and Dunja Šešelja2 July 18-19, RUB, Bochum

• 1. Faculty of Economics and Social Sciences, Heidelberg University
• 2. Institute for Philosophy II, Ruhr-University Bochum

Zollman’s 2010 model Changing some assumptions. . . Our results Conclusion

1/34

Zollman’s 2010 model

Zollman’s ABM

Modeling science by "bandit problems" A gambler, confronted with slot machines that have different

• bjective probabilities of success.

2/34

Zollman’s ABM

Modeling science by "bandit problems" A gambler, confronted with slot machines that have different

• bjective probabilities of success.

The research question How do different social structures impact the efficiency of scientific inquiry?

2/34

• pulling a bandit’s arm → testing a hypothesis
• the payoff of a slot machine → a successful application of a

given hypothesis/method/theory

• the objective probability of success (OPS) of a slot machine →

OPS of the given hypothesis/method/theory.

3/34

Modeling science by "bandit problems"

• scientists are presented with the choice between two

hypotheses

• they always choose to pursue the hypothesis that seems to be

more successful

• they update their beliefs via Bayesian reasoning (via beta

distributions), based on:

• 1. their own success
• 2. the success of some other agents, with whom they are

connected in a social network

4/34

Restricting the information flow

• unrestricted information flow appears to be harmful
• the cycle scores the best, then the wheel, and then the

complete graph

5/34

Changing some assumptions. . .

Static vs. dynamic epistemic success

5/34

Static epistemic success

Zollman’s parameters

• OPS(True theory)=0.5
• OPS(False theory)=0.499
• Success: converging onto the true theory.
• Scientists may never get it right.

6/34

Static epistemic success

Zollman’s parameters

• OPS(True theory)=0.5
• OPS(False theory)=0.499
• Success: converging onto the true theory.
• Scientists may never get it right.
• However. . .
• We observe that scientists eventually do get it right!
• The question is not if, but rather when.

6/34

Static epistemic success

Zollman’s parameters

• OPS(True theory)=0.5
• OPS(False theory)=0.499
• Success: converging onto the true theory.
• Scientists may never get it right.
• However. . .
• We observe that scientists eventually do get it right!
• The question is not if, but rather when.

Dynamic notion of success

• OPS = probability of gaining corroborating evidence given the

current state of inquiry: Current probability of success

6/34

7/34

"Restless bandit"

8/34

"Restless bandit"

9/34

Introducing dynamic epistemic success

Current probability of success (CPS)

• agent on True theory: CPS(T) moves 0.1% towards 1.
• agent on False theory: CPS(T) moves 0.1% towards 0.

More precisely:

• APS(True theory)=1 and APS(False theory)=0
• CPSnew(T) = CPSold(T) + f (d)
• f (d) = d/s
• d = APS(T) − OPS(T)

10/34

Basic notions

Zollman (2010) Our ABM SPS(T) SPS(T) Updates: beta distribution Updates: beta distribution OPS(T) – static CPS(T) – dynamic APS(T) ∈ {0, 1} – static

11/34

(The lack of) Critical interaction

11/34

Lack of critical interaction

Critical aspect of interaction: not represented neither explicitly, nor implicitly Epistemic benefits of criticism criticism exposes errors in research

12/34

Critical interaction

13/34

Introducing critical interaction

Assumptions

• criticism is truth conducive (Moffett (2007); Betz (2012))
• it occurs between proponents of rivaling theories

14/34

Introducing critical interaction

Assumptions

• criticism is truth conducive (Moffett (2007); Betz (2012))
• it occurs between proponents of rivaling theories

Triggering condition:

• every time an agent pursuing Tx receives information from an

agent pursuing Ty, such that Ty turns out to be better than she had expected: SPSold(Ty) < SPSnew(Ty). Critical interaction

• agent on True theory: CPS(T) moves 0.1% towards 1.
• agent on False theory: CPS(T) moves 0.1% towards 0.

14/34

Aggregation problem

14/34

Scientists can distinguish between similarly successful theories

Zollman’s parameters

• OPS(True theory)=0.5
• OPS(False theory)=0.499

Agents can perfectly determine which theory is better, no matter how similarly successful their applications are. Aggregation problem Often it is impossible for scientists to say which theory is more worthy of pursuit, if they are similarly successful.

15/34

Treating similar theories as equally good

Threshold: The rival theory counts as better only if it surpasses one’s own theory by the margin of 0.1.

16/34

(The lack of) Inertia

16/34

Scientists easily abandon their inquiry

• agents are easily swayed by new evidence
• they abandon their theory if only the evidence suggests the

rival is superior Rational inertia in one’s inquiry Taking time to examine whether problems can be resolved.

17/34

Inertia of one’s inquiry

Jump threshold: Agents switch theories only after the rival has turned out to be - better for a certain number of rounds (e.g. 10 rounds).

18/34

Our results

Parameters

• 10.000 runs for each data point
• each simulation: up to 100.000 rounds
• A run stops when all agents converge on the better theory for

the final time.

• We measure how much time steps they need to do so.

19/34

Difficult inquiry (improvements in inquiry happen rarely)

19/34

Capturing Zollman’s (2010) results with dynamic CPS

1000 2000 3000 4000 Average Round Converged 2 4 6 8 10 12 scientists cycle complete

Figure 1: No critical interaction, no theory threshold, no jump threshold; global improvement in CPS every 100 round.

20/34

Adding critical interaction

1000 2000 3000 4000 Average Round Converged 2 4 6 8 10 12 scientists cycle complete

Figure 2: Critical interaction, no theory threshold, no jump threshold; global improvement in CPS every 100 round.

21/34

Adding jump threshold

500 1000 1500 2000 Average Round Converged 2 4 6 8 10 12 scientists cycle complete

Figure 3: No critical interaction, no theory threshold, jump threshold

• f 10; global improvement in CPS every 100 round.

22/34

Difficult inquiry, no theory threshold

500 1000 1500 2000 Average Round Converged 2 4 6 8 10 12 scientists cycle complete

Figure 4: Critical interaction, no theory threshold, jump threshold of 10; global improvement in CPS every 100 round.

23/34

Let’s make inquiry even more difficult. . . (in terms of theory threshold)

23/34

Adding theory threshold (difficult inquiry)

10000 20000 30000 Average Round Converged 2 4 6 8 10 12 scientists cycle complete

Figure 5: No critical interaction, theory threshold of 0.1, no jump threshold; global improvement in CPS every 100 round.

24/34

Adding critical interaction (difficult inquiry)

1000 2000 3000 4000 Average Round Converged 2 4 6 8 10 12 scientists cycle complete

Figure 6: Critical interaction, theory threshold of 0.1, no jump threshold; global improvement in CPS every 100 round.

25/34

Adding jump threshold (difficult inquiry)

1000 2000 3000 4000 Average Round Converged 2 4 6 8 10 12 scientists cycle complete

Figure 7: Critical interaction, theory threshold of 0.1, jump threshold of 10; global improvement in CPS every 100 round.

26/34

Summing up

Difficult inquiry

• Efficiency is obtained by either critical interaction or by being

cautious.

• The degree of connectedness doesn’t always play a major role.

27/34

Conclusion

• Results of Zollman’s 2010 model are not robust under different

assumptions concerning scientific inquiry

• Further research:
• different decision making procedures
• sensitivity analysis
• etc.

28/34

When are models informative of real world phenomena?

29/34

Thank you!

30/34

Easy inquiry

20 40 60 Average Round Converged 2 4 6 8 10 12 scientists cycle complete

Figure 8: No critical interaction, no theory threshold, no jump threshold; global improvement in CPS every 10 rounds 1%.

Bibliography

Bibliography i

References

Betz, G.: 2012, Debate dynamics: How controversy improves our beliefs, Vol. 357. Springer Science & Business Media. Douglas, H. E.: 2009, Science, Policy, and the Value-Free Ideal. University of Pittsburgh Press. Goldman, A. and T. Blanchard: 2016, ‘Social Epistemology’. In:

• E. N. Zalta (ed.): The Stanford Encyclopedia of Philosophy.

Stanford University, summer 2016 edition.

Bibliography ii

Grim, P.: 2009, ‘Threshold Phenomena in Epistemic Networks.’. In: AAAI Fall Symposium: Complex Adaptive Systems and the Threshold Effect. pp. 53–60. Grim, P., D. J. Singer, S. Fisher, A. Bramson, W. J. Berger, C. Reade, C. Flocken, and A. Sales: 2013, ‘Scientific networks on data landscapes: question difficulty, epistemic success, and convergence’. Episteme 10(04), 441–464. Martini, C. and M. F. Pinto: 2016, ‘Modeling the social

• rganization of science’. European Journal for Philosophy of

Science pp. 1–18. Moffett, M.: 2007, ‘Reasonable disagreement and rational group inquiry’. Episteme: A Journal of Social Epistemology 4(3), 352–367.

Bibliography iii

Rosenstock, S., C. O’Connor, and J. Bruner: 2016, ‘In Epistemic Networks, is Less Really More?’. Philosophy of Science. Wray, K. B.: 2011, Kuhn’s evolutionary social epistemology. Cambridge University Press. Zollman, K. J. S.: 2007, ‘The communication structure of epistemic communities’. Philosophy of Science 74(5), 574–587. Zollman, K. J. S.: 2010, ‘The epistemic benefit of transient diversity’. Erkenntnis 72(1), 17–35.