Peer Prediction Mechanisms and their Connections to Machine Learning - - PowerPoint PPT Presentation

▶

Jun 20, 2023 664 likes •833 views

Peer Prediction Mechanisms and their Connections to Machine Learning Jens Witkowski ETH Zurich Game Theory Meets Computational Learning Theory Dagstuhl June 22, 2017 Joint work with David C. Parkes (Harvard) and Rafael Frongillo

SLIDE 1

Peer Prediction Mechanisms and their Connections to Machine Learning

Jens Witkowski∗ ETH Zurich Game Theory Meets Computational Learning Theory Dagstuhl June 22, 2017

∗Joint work with David C. Parkes (Harvard) and Rafael Frongillo (Boulder).

SLIDE 2

Forecasting Peer Prediction Connections to ML Conclusion

Proper Scoring Rules

Forecaster with belief p ∈ [0, 1] reports belief y ∈ [0, 1].

March 1: pay R(y, ω), where ω = 1, if event occurs 0, if not. If R(y, ω) proper, Ep[R(y, ω)] maximized by reporting y = p. Example: Quadratic Scoring Rule [Brier, 1950] Rq(y, ω) = 1 − (y − ω)2

1 / 14

SLIDE 3

Forecasting Peer Prediction Connections to ML Conclusion

“Effective” Scoring Rules

What if allowed reports y are restricted? True belief: p = 0.7. Allowed reports: y1 = 0.6 or y2 = 0.79. p 1 y1 =0.6 y2 =0.79 Quadratic Rule: report y “closest” to p! Theorem [Friedman, 1983] The Quadratic Rule is effective: if Y ⊆ [0, 1], reporting min

y∈Y |y − p| maximizes expected score.

2 / 14

SLIDE 4

Forecasting Peer Prediction Connections to ML Conclusion

Motivation

Objective: Truthful elicitation of opinions or experiences.

3 / 14

SLIDE 5

Forecasting Peer Prediction Connections to ML Conclusion

Peer Prediction

Elicit informative signal (e.g. “high” or “low” experience). Ground truth never observed. Pay agent given her report xi and report of other agent xj. Objective: truthful reporting of signal is a Bayes NE. Key: Agent i’s signal informative about j’s signal. i j Si Sj i j Si = si Sj = sj

4 / 14

SLIDE 6

Forecasting Peer Prediction Connections to ML Conclusion

Belief Model

bad

l

good

h 0.9 0.1 0.4 0.6 0.7 0.3 i j Sj =h Belief that Sj = h: p(h) = 0.25 p(h|h) = 0.46 p(h|l) = 0.18

5 / 14

SLIDE 7

Forecasting Peer Prediction Connections to ML Conclusion

Simplest Mechanism: Output Agreement

Compare two agents’ reports and pay:

if reports agree, $0

therwise.

Example with Si = h i j

Sj =h

= 0.46 E

payment|xi = h
= $0.92

i j

Sj =l

= 0.54 E

payment|xi = l
= $1.08

Output Agreement not truthful for this belief model!

6 / 14

SLIDE 8

Forecasting Peer Prediction Connections to ML Conclusion

Classical Peer Prediction [Miller et al., 2005]

i xi = h

p(h|l)=0.18 p(h|h)=0.46

j xj R

0.46, xj
Truthful if p(h|h) = p(h|l)!

Intuition

Define agent j’s signal report as event.

Restrict possible belief reports to possible posteriors. Crucial: mechanism knows belief model!

7 / 14

SLIDE 9

Forecasting Peer Prediction Connections to ML Conclusion

Shadowing Method [W. and Parkes, 2012]

Assumption: only y ∈

p(h|l), p(h|h)
known.

y

i xi = h

y δ δ

Rq

y+δ, xj
j

xj Truthful: agent prefers “shadow posterior” closer to true belief! Crucial: quadratic scoring rule Rq!

8 / 14

SLIDE 10

Forecasting Peer Prediction Connections to ML Conclusion

Shadowing Method: Key Idea

Challenge: no knowledge of posteriors p(h|h) or p(h|l)! Compute “Shadow posteriors” y + δ and y − δ: y 1 p(h|l) p(h|h) δ δ Observe: Si = h ⇒ y + δ closer to p(h|h) than y − δ. Si = l ⇒ y − δ closer to p(h|l) than y + δ.

9 / 14

SLIDE 11

Forecasting Peer Prediction Connections to ML Conclusion

1. Learning Mechanisms from Reports

Can y be learned from data? Idea Sketch Known: p(h) ∈

p(h|l), p(h|h)
.

Consider other questions/tasks with same prior. Empirical frequency of x = h reports on those will be close to p(h) with high probability. Main Result [W. and Parkes, 2013] The Shadowing Method with y = ˆ pi(h) is strictly truthful given enough samples where this number depends on a lower bound

n the belief change from prior to posterior.

10 / 14

SLIDE 12

Forecasting Peer Prediction Connections to ML Conclusion

2. Peer Prediction Mechanisms are Loss Functions

Peer Prediction Mechanism xi xj h l h 1 l 1 Output Agreement xi xj h l h 2 l 1 2 Shadowing w/ y = 2

3

Classification Loss sign(wTxi) yi +1 −1 +1 1 −1 1 0/1 Loss sign(wTxi) yi +1 −1 +1 2 −1 1 Cost-sensitive Loss

11 / 14

SLIDE 13

Forecasting Peer Prediction Connections to ML Conclusion

Scoring Rules for Properties

Theorem [Frongillo and W., 2017] Peer Prediction Mechanisms are equivalent to scoring rules for properties of probability distributions. Example: Output Agreement a b c arg maxxi Pr(Sj = xi|Si = si) a b c arg maxyi Pr(Yi = yi|xi, w) Output Agreement elicits the mode.

12 / 14

SLIDE 14

Forecasting Peer Prediction Connections to ML Conclusion

Conclusion

Peer Prediction mechanisms truthfully elicit private

pinions or experiences.

Connections between PP mechanisms and ML:

Learn truthful mechanisms using reports on other items.

Mechanisms are equivalent to loss functions eliciting property of conditional label probability.

THANK YOU!

13 / 14

SLIDE 15

References I

Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review, 78(1):1–3. Friedman, D. (1983). Effective Scoring Rules for Probabilistic Forecasts. Management Science, 29(4):447–454. Frongillo, R. and Witkowski, J. (2017). A geometric perspective on minimal peer prediction. ACM Transactions on Economics and Computation (TEAC). Forthcoming.

13 / 14

SLIDE 16

References II

Miller, N., Resnick, P ., and Zeckhauser, R. (2005). Eliciting Informative Feedback: The Peer-Prediction Method. Management Science, 51(9):1359–1373. Witkowski, J. (2014). Robust Peer Prediction Mechanisms. PhD thesis, Department of Computer Science, Albert-Ludwigs-Universität Freiburg. Witkowski, J. and Parkes, D. C. (2012). A Robust Bayesian Truth Serum for Small Populations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI’12), pages 1492–1498.

14 / 14