Eliciting Informative Feedback: The Peer-Prediction Method Colin - - PowerPoint PPT Presentation

1

Eliciting Informative Feedback: The Peer-Prediction Method

Colin Zheng and Kenneth Wang

2 2

Rater 1

Camera Quality: dist 𝐸 over {𝑀, 𝐼}

Rater 2 𝑡2~𝑇 𝑡1~𝑇

𝑇: dist over 𝑚, ℎ jointly w/ 𝐸

Center:

• Ebay
• NetFlix
• Amazon

Report 𝑠1 Report 𝑠2

The Setting

3 3

Rater 1

Camera Quality: dist 𝐸 over {𝑀, 𝐼}

Rater 2 𝑡2~𝑇 𝑡1~𝑇

𝑇: dist over 𝑚, ℎ jointly w/ 𝐸

Center:

• Ebay
• NetFlix
• Amazon

Report 𝑠1 Report 𝑠2

The Setting

Task:

How to make rational raters report honestly (𝑠𝑗 = 𝑡𝑗)

Naive Attempt: reward 𝜐 = $1 if 𝑠1 = 𝑠2, 𝜐 = $0 otherwise Problem: rater 1 will report “more likely signal” of rater 2

4 4

Rater 1

Camera Quality: dist 𝐸 over {𝑀, 𝐼}

Rater 2 𝑡2~𝑇 𝑡1~𝑇

𝑇: dist over 𝑚, ℎ jointly w/ 𝐸

Center:

• Ebay
• NetFlix
• Amazon

Report 𝑠1 Report 𝑠2

The Setting

Task (formulation):

Reporting 𝑠𝑗 = 𝑡𝑗 is a Nash Equilibrium ∀ rater 𝑗, signal 𝑛, 𝑦 ≠ 𝑛 𝔽𝑡2 𝜐1 𝑡1, 𝑡2 𝑡1 = 𝑛 ≥ 𝔽𝑡2 𝜐1 𝑦, 𝑡2 𝑡1 = 𝑛 𝔽𝑡1 𝜐2 𝑡2, 𝑡1 𝑡2 = 𝑛 ≥ 𝔽𝑡1 𝜐2 𝑦, 𝑡1 𝑡2 = 𝑛

5 5

Rater 1

Camera Quality: dist 𝐸 over {𝑀, 𝐼}

Rater 2 𝑡2~𝑇 𝑡1~𝑇

𝑇: dist over 𝑚, ℎ jointly w/ 𝐸

Center:

• Ebay
• NetFlix
• Amazon

Report 𝑠1 Report 𝑠2

Solution using Proper Scoring Rule

Task:

choose payoff 𝜐1 such that 𝔽𝑡2 𝜐1 𝑦, 𝑡2 𝑡1 = 𝑛 is maximized at 𝑦 = 𝑛 Equivalently, 𝔽𝑨~𝑎 𝜐1 𝑦, 𝑨 where 𝑎 = 𝑡2 𝑡1=𝑛

Recall: 𝑔 is Proper Scoring Rule if

𝔽𝑨~𝑎[𝑔(𝑄, 𝑨)] is maximized at 𝑄 = 𝑎

So let 𝜐1 𝑦, 𝑨 = 𝑔(𝑡2 𝑡1=𝑦,𝑨)

6 6

Issue: costs to the rater

In reality,

• Evaluating and reporting honestly incur a

relative cost 𝑑 > 0

(e.g. testing is time consuming, opportunity cost, ...)

• Can be offset by scaling up the payoffs
• Problems:

– Paying too much for truthful information? – What if 𝑑 is unknown? [PRGJ08]

7 7

Issue: costs to the center

To reduce center’s cost:

• Budget balancing:

– pair up raters, make each pair a zero-sum game

• Use linear optimization to find cost-optimal

payoff function [JF06]

8 8

Issue: risk aversion

• When the center knows the raters’ utility

function(s)

• When the center does not know the utility

function(s)

• Using multiple reference raters reduces risk

9 9

Issue: collusion

• Using an honest reference rater
• Randomized reference rater
• Outside experts...

10 10

Issue: Unknown/diff. priors

• In reality, raters have
• diff. beliefs about

(𝐸, 𝑇), unknown to the center

• Addressed in [WP12]

for binary signals 𝑇

Rater 1

Camera Quality: dist 𝐸 over {𝑀, 𝐼}

Rater 2 𝑡2~𝑇 𝑡1~𝑇 𝑇: dist over

𝑚, ℎ jointly w/ 𝐸

Center:

• Ebay
• NetFlix
• Amazon

Report 𝑠1 Report 𝑠2