Surrogate Scoring Rules Juntao Wang, Harvard University Yiling - - PowerPoint PPT Presentation

Ø Research Question: Can we incentivize high-quality prediction

when the ground truth is unavailable?

Ø Our Answer: Yes! Surrogate Scoring Rules (SSR) Ø A Motivation Example:

• A principal cares “How likely a study can be replicated?”
• Forecasters are asked to provide a probabilistic prediction.
• The SCORE program crowdsourced this question for 3000 studies to

hundreds of researchers, while only 300 will be put into real replication test.

Surrogate Scoring Rules

Yang Liu, UCSC Juntao Wang, Harvard University Yiling Chen, Harvard University

Implies 2 desirable properties

Ø Incentivize truthful reporting (P1) Ø Accurate forecasts get higher expected rewards (P2)

Strictly Proper Scoring Rules SSR SSR Mechanism Roadmap.

Access to ground truth Access to a noisy ground truth No access to ground truth

Building block Building block

Strictly proper scoring rules (SPSR) (existing work):

! "#, %

Report of agent & Ground truth % ∈ {0,1} Truthfulness property (P1) Accuracy property (P2): Example:

• Let ,∗ = true distribution of %
• For each !, ∃ divergence function /:

01~3∗ ! "#, % = 56789 − /(,∗||"#) ! "#, % = 1 − "# − % >

Surrogate Scoring Rules (SSR):

?("#, @; BC

D, BC E) @ – A noisy ground truth with error rates:

• BC

D: = Pr @ = 0 % = 1 (known)

• BC

E: = Pr @ = 1 % = 0 (known)

Unbiasedness property

• Enables us to inherit P1, P2 from SPSR
• Def. ? "#, @

is SSR if and only if ∀"#, 0J ? "#, % = 01 ! "#, % Implementation:

? "#, @ = 1 = 1 − BJ

E ⋅ ! "#, 1 − BC D ⋅ !("#, 0)

1 − BJ

E − BJ D

? "#, @ = 0 = 1 − BJ

D ⋅ ! "#, 0 − BC E ⋅ ! "#, 1

1 − BJ

E − BJ D

• Def. ! "#, %

is SPSR if and only if ∀,# ≠ "#, 01~3M ! ,#, % > 01~3M ! "#, %

SSR Mechanisms

3

SSR Mechanism. When score !",$ for agent %, apply SSR:

& !",$, '; )*

+, )* ,

Ø Construct a noisy ground truth ':

“For a task -, uniformly randomly pick an agent . ≠ %, draw ' = 1 with probability !2,$”

Ø Estimate the error rates: “Methods of Moments” Ø Apply SSR against ' and )*

+, )* ,

1 2 3 1 2 3 Main Theorem Under A1~A4, in SSR mechanisms, truthful reporting is a uniform dominant strategy for M, N → ∞, or for arbitrarily discretized report space. Unbiasedness of SSR mechanism to SPSR 789:;. & !",$, ' = = !",$, >

$ + @ 1

A + 1 B Setting:

• A set of agents A (index %)
• A set of tasks B (index -)
• Ground truth >

$ ∈ {0,1}

• Belief G",$ of agents on tasks
• Report !",$ from agents on tasks
• A task is assigned to at least 3

agents Assumptions:

• A1. Tasks are homogeneous and

independent

• A2. Beliefs are independent

conditioned on >

$

• A3. Uniform strategy across tasks
• A4. The principal knows

H #>

$ = 1 > #> $ = 0 .

Experimental Evaluation

– – on 14 real-world human forecast datasets

Ø SSR is close to true score (SPSR) Ø SSR has strongest correction to true scores SPSR than others Ø Deployed in RM! (RM blog)

Paper: https://arxiv.org/abs/1802.09158 Contact: juntaowang@g.harvard.edu