Robust Bayesian Truth Serum Presentation by Mark Bun and Bo Waggoner - - PowerPoint PPT Presentation

▶

Oct 17, 2022 136 likes •975 views

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Robust Bayesian Truth Serum Presentation by Mark Bun and Bo Waggoner 2012-12-03 Presentation by Mark Bun and Bo Waggoner Robust

SLIDE 1

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms

Robust Bayesian Truth Serum

Presentation by Mark Bun and Bo Waggoner 2012-12-03

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 2

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms

Outline

1

Introduction and Setting Recap: Human Computation Mechanisms so far The RBTS Approach Setting

2

RBTS and Shadowing Shadowing RBTS

3

PP Without Common Prior

4

Summary: Human Computation Mechanisms Assumptions and Results

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 3

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Recap

Peer prediction: Elicit? Rewards?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 4

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Recap

Peer prediction: Elicit? Rewards? Bayesian Truth Serum: Elicit? Rewards?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 5

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Big problem with implementing PP?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 6

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Big problem with implementing PP? Mechanism needs to know the information structure!

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 7

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Big problem with implementing PP? Mechanism needs to know the information structure! Big problem with BTS?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 8

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Big problem with implementing PP? Mechanism needs to know the information structure! Big problem with BTS? Requires n → ∞!

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 9

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Big problem with implementing PP? Mechanism needs to know the information structure! Big problem with BTS? Requires n → ∞! Other problems with BTS? Does RBTS resolve them?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 10

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Goal: Truthful mechanism for any n that doesn’t rely on the mechanism knowing the information structure.

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 11

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Goal: Truthful mechanism for any n that doesn’t rely on the mechanism knowing the information structure. Approach: Recall: Elicit two-part report (prediction and signal) Which one is easy to incentivize and which is difficult?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 12

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Goal: Truthful mechanism for any n that doesn’t rely on the mechanism knowing the information structure. Approach: Recall: Elicit two-part report (prediction and signal) Which one is easy to incentivize and which is difficult? To elicit the signal: Use shadowing!

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 13

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Belief system

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 14

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Belief system

Consists of state T drawn from {1, . . . , m} and signals S drawn from {l, h} = {0, 1}

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 15

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Belief system

Consists of state T drawn from {1, . . . , m} and signals S drawn from {l, h} = {0, 1} Agents have common prior Pr[T = t] and Pr[S = h|T = t].

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 16

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Belief system

Consists of state T drawn from {1, . . . , m} and signals S drawn from {l, h} = {0, 1} Agents have common prior Pr[T = t] and Pr[S = h|T = t]. “Impersonally informative”

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 17

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Belief system

Consists of state T drawn from {1, . . . , m} and signals S drawn from {l, h} = {0, 1} Agents have common prior Pr[T = t] and Pr[S = h|T = t]. “Impersonally informative” : For every i, j, k, write p{si} = Pr[Sj = h|Si = si]

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 18

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Belief system

Consists of state T drawn from {1, . . . , m} and signals S drawn from {l, h} = {0, 1} Agents have common prior Pr[T = t] and Pr[S = h|T = t]. “Impersonally informative” : For every i, j, k, write p{si} = Pr[Sj = h|Si = si] p{si,sj} = Pr[Sk = h|Si = si, Sj = sj]

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 19

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Admissibility

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 20

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Admissibility

Admissible prior:

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 21

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Admissibility

Admissible prior:

1 At least two states (m ≥ 2) Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 22

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Admissibility

Admissible prior:

1 At least two states (m ≥ 2) 2 Every state has positive probability (Pr[T = t] > 0) Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 23

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Admissibility

Admissible prior:

1 At least two states (m ≥ 2) 2 Every state has positive probability (Pr[T = t] > 0) 3 Assortative property:

Pr[S = h|T = 1] < · · · < Pr[S = h|T = m]

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 24

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Admissibility

Admissible prior:

1 At least two states (m ≥ 2) 2 Every state has positive probability (Pr[T = t] > 0) 3 Assortative property:

Pr[S = h|T = 1] < · · · < Pr[S = h|T = m]

4 Fully mixed: 0 < Pr[S = h|T = t] < 1 Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 25

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Recap: Human Computation Mechanisms so far The RBTS Approach Setting

Admissibility

Admissible prior:

1 At least two states (m ≥ 2) 2 Every state has positive probability (Pr[T = t] > 0) 3 Assortative property:

Pr[S = h|T = 1] < · · · < Pr[S = h|T = m]

4 Fully mixed: 0 < Pr[S = h|T = t] < 1

Lemma 6: For an admissible prior, 1 > p{h,h} > p{h} > p{h,l} = p{l,h} > p{l} > p{l,l} > 0

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 26

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

What does shadowing solve?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 27

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

What does shadowing solve?

Truthfully elicit signals instead of beliefs

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 28

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

What does shadowing solve?

Truthfully elicit signals instead of beliefs ω ∈ {0, 1} a binary future event

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 29

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

What does shadowing solve?

Truthfully elicit signals instead of beliefs ω ∈ {0, 1} a binary future event Agent i draws a signal Si ∈ {0, 1} = {l, h}

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 30

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

What does shadowing solve?

Truthfully elicit signals instead of beliefs ω ∈ {0, 1} a binary future event Agent i draws a signal Si ∈ {0, 1} = {l, h} How do we get agent i to truthfully reveal Si?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 31

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Na¨ ıve approach

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 32

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Na¨ ıve approach

1 Choose a strictly proper binary scoring rule R(y, ω) Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 33

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Na¨ ıve approach

1 Choose a strictly proper binary scoring rule R(y, ω) 2 Ask agent i for signal report xi ∈ {0, 1} using R(xi, ω) Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 34

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Na¨ ıve approach

1 Choose a strictly proper binary scoring rule R(y, ω) 2 Ask agent i for signal report xi ∈ {0, 1} using R(xi, ω)

Might not work if i’s beliefs p{Si} about ω are far from xi!

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 35

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Solution: Use a reference prediction

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 36

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Solution: Use a reference prediction

1 Let y ∈ (0, 1) be an arbitrary prediction report Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 37

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Solution: Use a reference prediction

1 Let y ∈ (0, 1) be an arbitrary prediction report 2 Set δ = min(y, 1 − y) Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 38

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Solution: Use a reference prediction

1 Let y ∈ (0, 1) be an arbitrary prediction report 2 Set δ = min(y, 1 − y) 3 Create a shadow report for i:

y′

i =

y + δ if xi = 1 y − δ if xi = 0

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 39

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Solution: Use a reference prediction

1 Let y ∈ (0, 1) be an arbitrary prediction report 2 Set δ = min(y, 1 − y) 3 Create a shadow report for i:

y′

i =

y + δ if xi = 1 y − δ if xi = 0

4 Score agent xi using the quadratic scoring rule Rq(y′

i , ω)

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 40

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

When and why does shadowing work?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 41

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

When and why does shadowing work?

Intuition: Truthfully reporting xi = Si pulls the reference prediction toward i’s posterior

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 42

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

When and why does shadowing work?

Intuition: Truthfully reporting xi = Si pulls the reference prediction toward i’s posterior Selten ’98: Let Y ⊂ [0, 1]. If agent i has beliefs p, then she maximizes her expected score Rq(y, ω) over y ∈ Y by minimizing |y − p|.

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 43

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

When and why does shadowing work?

Intuition: Truthfully reporting xi = Si pulls the reference prediction toward i’s posterior Selten ’98: Let Y ⊂ [0, 1]. If agent i has beliefs p, then she maximizes her expected score Rq(y, ω) over y ∈ Y by minimizing |y − p|. Lemma 8: Suppose agent i has observed signals I ∈ {l, h}k and Si. If p{l,I} < y < p{h,I}, then agent i should truthfully report xi = Si.

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 44

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

What’s missing?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 45

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

What’s missing?

How do we pick y to guarantee p{l,I} < y < p{h,I}?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 46

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

What’s missing?

How do we pick y to guarantee p{l,I} < y < p{h,I}? How do we pick ω when there is no ground truth?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 47

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Robust Bayesian Truth Serum

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 48

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Robust Bayesian Truth Serum

Idea: Shadowing + reference raters

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 49

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Robust Bayesian Truth Serum

Idea: Shadowing + reference raters Information report: xi ∈ {0, 1} represents agent i’s signal

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 50

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Robust Bayesian Truth Serum

Idea: Shadowing + reference raters Information report: xi ∈ {0, 1} represents agent i’s signal Prediction report: yi ∈ [0, 1] represents agent i’s prediction for the frequency of h signals

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 51

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Scoring

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 52

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Scoring

For each agent i, pick reference raters j, k

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 53

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Scoring

For each agent i, pick reference raters j, k Set y = yj and ω = xk.

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 54

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Scoring

For each agent i, pick reference raters j, k Set y = yj and ω = xk. Create a shadow report for i: y′

i =

yj + δ if xi = 1 yj − δ if xi = 0 where δ = min(yj, 1 − yj)

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 55

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Scoring

For each agent i, pick reference raters j, k Set y = yj and ω = xk. Create a shadow report for i: y′

i =

yj + δ if xi = 1 yj − δ if xi = 0 where δ = min(yj, 1 − yj) RBTS score: Rq(y′

i , xk)

information score

+ Rq(yi, xk)

prediction score

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 56

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Incentive compatibility

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 57

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Incentive compatibility

If n ≥ 3 agents have signals drawn from an admissible common prior, then reporting truthfully is a strict Bayes-Nash equilibrium.

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 58

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Incentive compatibility

If n ≥ 3 agents have signals drawn from an admissible common prior, then reporting truthfully is a strict Bayes-Nash equilibrium. Why should agents report their predictions truthfully?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 59

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Incentive compatibility

If n ≥ 3 agents have signals drawn from an admissible common prior, then reporting truthfully is a strict Bayes-Nash equilibrium. Why should agents report their predictions truthfully? How about signals?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 60

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Incentive compatibility

If n ≥ 3 agents have signals drawn from an admissible common prior, then reporting truthfully is a strict Bayes-Nash equilibrium. Why should agents report their predictions truthfully? How about signals? Enough to show that admissibility = ⇒ p{l,I} < yj < p{h,I}.

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 61

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Incentive compatibility

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 62

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Incentive compatibility

Lemma 6: For an admissible prior, 1 > p{h,h} > p{h} > p{h,l} = p{l,h} > p{l} > p{l,l} > 0

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 63

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Incentive compatibility

Lemma 6: For an admissible prior, 1 > p{h,h} > p{h} > p{h,l} = p{l,h} > p{l} > p{l,l} > 0 Case 1: yj = p{h} = ⇒ p{l,h} < yj = p{h} < p{h,h}

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 64

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Incentive compatibility

Lemma 6: For an admissible prior, 1 > p{h,h} > p{h} > p{h,l} = p{l,h} > p{l} > p{l,l} > 0 Case 1: yj = p{h} = ⇒ p{l,h} < yj = p{h} < p{h,h} Case 2: yj = p{l} = ⇒ p{l,l} < yj = p{l} < p{h,l}

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 65

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Incentive compatibility

Lemma 6: For an admissible prior, 1 > p{h,h} > p{h} > p{h,l} = p{l,h} > p{l} > p{l,l} > 0 Case 1: yj = p{h} = ⇒ p{l,h} < yj = p{h} < p{h,h} Case 2: yj = p{l} = ⇒ p{l,l} < yj = p{l} < p{h,l} Recap: p{l,Sj} < yj < p{h,Sj}, so agent i should truthfully reveal xi = Si

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 66

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Properties of RBTS

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 67

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Properties of RBTS

Works for any number of agents n ≥ 3

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 68

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Properties of RBTS

Works for any number of agents n ≥ 3 Scores are well-defined between 0 and 2.

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 69

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Properties of RBTS

Works for any number of agents n ≥ 3 Scores are well-defined between 0 and 2. Participation is “ex post individually rational”.

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 70

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Properties of RBTS

Works for any number of agents n ≥ 3 Scores are well-defined between 0 and 2. Participation is “ex post individually rational”. Numerically robust

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 71

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Extensions

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 72

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Extensions

Adapting to a budget constraint.

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 73

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Extensions

Adapting to a budget constraint. Scaling, randomized exclusion.

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 74

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Extensions

Adapting to a budget constraint. Scaling, randomized exclusion. More than two outcomes?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 75

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Shadowing RBTS

Extensions

Adapting to a budget constraint. Scaling, randomized exclusion. More than two outcomes? Other proper scoring rules?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 76

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms

Peer Prediction Without a Common Prior. Witkowski, Parkes. EC-2012.

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 77

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms

Peer Prediction Without a Common Prior. Witkowski, Parkes. EC-2012. Same setting as RBTS except each agent may have a different information structure.

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 78

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms

Peer Prediction Without a Common Prior. Witkowski, Parkes. EC-2012. Same setting as RBTS except each agent may have a different information structure. Elicit both the agent’s prior and posterior probability that another agent will get a high signal.

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 79

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms

Peer Prediction Without a Common Prior. Witkowski, Parkes. EC-2012. Same setting as RBTS except each agent may have a different information structure. Elicit both the agent’s prior and posterior probability that another agent will get a high signal. Infer whether agent’s signal was high or low

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 80

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms

Peer Prediction Without a Common Prior. Witkowski, Parkes. EC-2012. Same setting as RBTS except each agent may have a different information structure. Elicit both the agent’s prior and posterior probability that another agent will get a high signal. Infer whether agent’s signal was high or low Score = score(prior) + score(posterior)

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 81

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms

Peer Prediction Without a Common Prior. Witkowski, Parkes. EC-2012. Same setting as RBTS except each agent may have a different information structure. Elicit both the agent’s prior and posterior probability that another agent will get a high signal. Infer whether agent’s signal was high or low Score = score(prior) + score(posterior) OR elicit prior and signal, and shadow to get a posterior (complicated)

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 82

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Assumptions and Results Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum

SLIDE 83

Introduction and Setting RBTS and Shadowing PP Without Common Prior Summary: Human Computation Mechanisms Assumptions and Results

PP PP-CP BTS RBTS common prior exists designer knows CP information structure # of players # of signals designer learns?

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum