SLIDE 1 1
Informed Truthfulness for Multi-Task Peer Prediction
Victor Shnayder, Arpit Agarwal, Rafael Frongillo, David C. Parkes
Nov 3, 2016; HCOMP’16 Foundations Workshop
Let’s talk about crowdsourcing
Informed Truthfulness for Multi-Task Peer Prediction Victor Shnayder - - PowerPoint PPT Presentation
Informed Truthfulness for Multi-Task Peer Prediction Victor Shnayder - - PowerPoint PPT Presentation
Informed Truthfulness for Multi-Task Peer Prediction Victor Shnayder , Arpit Agarwal, Rafael Frongillo, David C. Parkes Nov 3, 2016; HCOMP16 Foundations Workshop 1 Lets talk about crowdsourcing 2 Task: How many cows in this image? 3
SLIDE 2 2
SLIDE 3
Task: How many cows in this image?
3 SLIDE 4
Count: 23
4 SLIDE 5
… or just guess: 42
5 SLIDE 6
Get feedback: “23 is good. $0.02”
6 SLIDE 7
Get feedback: “42 is way off! $0.00”
7 SLIDE 8
Get feedback: “42 is way off! $0.00” Where did feedback come from?
8 SLIDE 9
Get feedback: “42 is way off! $0.00” Where did feedback come from?
Gold standard: professional cow counter said 23
9 SLIDE 10
Get feedback: “42 is way off! $0.00” Where did feedback come from?
Machine vision: automated cow counter said 20-25
10 SLIDE 11
Get feedback: “42 is way off! $0.00” Where did feedback come from?
Peers: 3 others said 20, 22, 24
11 SLIDE 12
Peer prediction: giving feedback based on peer reports
12 SLIDE 13
Evaluate
“7 cows” “good!” “1 cow” “1 cow” “good!” “bad!” Observe Observe Observe
13 SLIDE 14
Applications beyond s
Gather location-specific info Image and video labeling Search result evaluation Academic peer review Participatory sensing Evaluations for peer assessment in massive courses
14 SLIDE 15
Goals:
Ensure truthful equilibrium exists and is attractive Impossibility results
15 SLIDE 16
Task model
Agent 1 Task i
16Signal: 1 … n
SLIDE 17
Task model
Agent 1 Task i
17Prior probability P(i)
SLIDE 18
Task model
Agent 1 Task Agent 2 i j
18Joint probability: P(i,j)
SLIDE 19
Task model
Agent 1 Task Agent 2 i j
19People can misreport!
SLIDE 20
Task model
Agent 1 Task Agent 2 i j
20Goal: design scores to encourage effort, truthful reports
SLIDE 21
Output agreement (von Ahn, Dabbish ‘04)
Agent 1 Task Agent 2
1 1
Agree! Score 1
21 SLIDE 22
Agent 1 Task Agent 2
1 2
Don’t agree: score 0
22Output agreement
SLIDE 23
Output agreement
Agent 1 Task Agent 2 Honest reporting is a correlated equil if my signal predicts yours
23 SLIDE 24
Output agreement
Agent 1 Task Agent 2
24To manipulate: all agents always report same thing
SLIDE 25
Ensuring truthful reporting is best
[Kamble et. al. ’15, Radanovic et. al. ’16]
Agent 1 Task Agent 2
1 1
Agree! Score 1/(scaling factor) Scaling factor learned from reports on many similar tasks. Truthfulness is an equilibrium, guarantees highest payoff.
25 SLIDE 26
Multi-task approach
[Dasgupta-Ghosh’13] Agent 1
Task 1 Task 2 1 1
26 SLIDE 27
Task 3
Agent 1
Task 1 Task 2
Agent 2
1 1 2 2
27Multi-task approach
[Dasgupta-Ghosh’13]
SLIDE 28
Task 3
Agent 1
Task 1 Task 2
Agent 2
1 2 1 2
Likely to match
28Key idea
SLIDE 29
Task 3
Agent 1
Task 1 Task 2
Agent 2
1 2 1 2
L e s s l i k e l y t
- m
a t c h
29Key idea
SLIDE 30
Task 3
Agent 1
Task 1 Task 2
Agent 2
1 2 1 2
Reward matching on shared tasks. Punish matching on non-shared tasks.
30Key idea
SLIDE 31
Task 3
Agent 1
Task 1 Task 2
Agent 2
1 2 1 2
31P(agree on shared) – P(agree on non-shared)
Why doesn’t this Just Work with >2 signals?
SLIDE 32
Our multi-signal mechanism: Correlated Agreement
- 1. Split tasks into shared and non-shared.
- 2. Score = (agree on shared) – (agree on non-shared)
“Agree” when reports aren’t equal, but positively correlated.
Constant or random reporting still has expected score 0
32 SLIDE 33
The Correlated Agreement mechanism
- 1. Informed truthful—being truthful is optimal, better
than constant or random reports
- 2. Works with minimal information with few tasks
- 3. Works with no information with many tasks
SLIDE 34
Connection to information theory
Truthful score ~ DTV(P(⋅,⋅) - P(⋅)P(⋅)) More agreement ⇒ higher scores. Other rules correspond to different distance functions. [Kong-Schoenebeck ‘16]
34 SLIDE 35
Open questions
Is peer prediction practical as primary incentive? When? Combine peer prediction with other incentive models in a single system? Heterogeneous agents Non-binary effort models Non-random task assignment (e.g. maps) Unintended correlated “signals”
35 SLIDE 36
Thank you!
shnayder@post.harvard.edu
36 SLIDE 37
Extra slides
37 SLIDE 38
Setup
Agents 1, 2, tasks k Signals i,j (require effort) Shared tasks, agent 1 tasks, agent 2 tasks Signal prior P(i), joint P(i,j) Strategies: F, G probability of reporting r given signal i. Informed strategy: depend on the signal somehow Truthful strategy: F*
38 SLIDE 39
Solution concepts
E(F, G): expected payment for a shared task (Strict) Proper: E(F*, G*) ≥ E(F, G*), for all F != F* Strong-truthful: E(F*, G*) ≥ E(F, G), for all F, G (if expected payment tied, F and G must be permutations) Informed-truthful: E(F*, G*) ≥ E(F, G), for all F, G (if expected payment tied, F and G must be informed)
39 SLIDE 40
A useful matrix
40 SLIDE 41
Example Delta
Joint:
41Delta: Sign(Delta): Prior:
SLIDE 42
Scoring matrix
42 SLIDE 43
Expected payment
43(Lemma: can restrict to deterministic strategies)
SLIDE 44
Key theorem The Correlated Agreement mechanism is informed truthful for all* models.
44 SLIDE 45
Proof sketch
45 SLIDE 46
Other results
Detail-free version of mechanism: learn scoring matrix from reports Correlated Agreement is maximal among large class of mechanisms that use this scoring structure Much simpler analysis of Dasgupta-Ghosh’13 mechanism
46 SLIDE 47
Binary mechanism (Dasgupta-Ghosh ’13):
- 1. Split tasks into shared and non-shared.
- 2. Score: (agree on shared) – (agree on non-shared)
Expected score: P(agree on shared) – P(agree on non-shared) Have to agree based on properties of shared task. Constant or random reporting has expected score = 0
47