Wisdom of The Crowd Lirong Xia Example: Crowdsourcing . . . . - - PowerPoint PPT Presentation

Lirong Xia

Wisdom of The Crowd

2

a b a b c Turker 1 Turker 2 Turker n

…

> >

Example: Crowdsourcing

. . . . . . . . . . . . . . . . . . . . . . . . . . . . > a b > b c >

The Condorcet Jury theorem. Ø Given

• two alternatives {O,M}.
• 0.5<p<1,

Ø Suppose

• each agent’s preferences is generated i.i.d., such that
• w/p p, the same as the ground truth
• w/p 1-p, different from the ground truth

Ø Then, as n→∞, the majority of agents’ preferences converges in probability to the ground truth

3

The Condorcet Jury theorem

[Condorcet 1785]

Pr( | ) = Pr( | ) = p>0.5

ØJustifies democracy and wisdom of the crowd

• “lays, among other things, the foundations of the

ideology of the democratic regime” [Paroush SCW-98]

4

Importance of the Jury Theorem

ØGroup competence

• Pr(maj(Pn)=a|a)
• Pn: n i.i.d. votes given ground truth a

ØRandom variable Xj : takes 1 w/p p, 0 otherwise

• encoding whether signal=ground truth

ØΣj=1nXj /nconverges to p in probability (Law of Large Numbers)

5

Proof

n j ! " # $ % & p j(1− p)n− j

j= n 2 ( ) ) * + + n

∑

ØGiven

• two alternatives {a,b}.
• competence 0.5<p<1,

ØSuppose

• agents’ signals are i.i.d. conditioned on the ground

truth

• w/p p, the same as the ground truth
• w/p 1-p, different from the ground truth
• agents truthfully report their signals

ØThe majority rule reveals ground truth as n→∞

6

Limitations of CJT

more than two? heterogeneous agents? dependent agents? strategic agents?

• ther rules?

ØDependent agents ØHeterogeneous agents ØStrategic agents ØMore than two alternatives

7

Extensions

8

An active area

Social Choice and Welfare American Political Science Review Games and Economic Behavior Mathematical Social Sciences Theory and Decision Public Choice Econometrica + JET

Myerson Shapley&Grofman

The group competence

• 1. is higher than that of any single agent
• Not always (same-vote equilibrium)
• 2. increases in the group size n
• Not always (same-vote equilibrium)
• 3. goes to 1 as n→∞
• Yes for some models and informative

equilibrium

9

Does CJT hold for strategic agents?

ØCommon interest Bayesian voting game [Austen-

Smith&Banks APSR-96]

• two alternatives {a, b}, two signals {A,B}, a prior,

Pr(signal|truth),

• pa=Pr(signal=A|truth=a)
• pb=Pr(signal=B|truth=b)
• agents have the same utility function U(outcome,

ground truth) =1 iff outcome = ground truth

• sincere voting: vote for the alternative with the highest

posterior probability

• informative voting: vote for the signal
• strategic voting: vote for the alternative with the

highest expected utility

10

Strategic voting

• 1. Nature chooses a ground truth g
• 2. Every agent j receives a signal sj~Pr(sj|g)
• 3. Every agent computes the posterior

distribution (belief) over the ground truth using Bayesian’s rule

• 4. Every agent chooses a vote to maximizes

her expected utility according to her belief

• 5. The outcome is computed by the voting rule

11

Timeline of the Bayesian game

ØTwo signals, two voters ØModel:

Pr( | ) = Pr( | ) = p>0.5

12

High level example

p 1-p + my vote , winner: utility for voting : half/half half/half p 1-p p 1-p Truthful agent: 1 0.5 0.5 Posterior: The other signal:

ØSetting

• Two alternatives {a, b}, two signals {A,B}
• Three agents
• Pr(A|a) = pa=0.6, Pr(B|b)=pb=0.8
• Prior: Pr(a)=0.2, Pr(b)=0.8

ØAn agent receives A

• Informative voting: a
• posterior probability:
• Pr(a|A) ∝Pr(a)×Pr(A|a) = 0.2×0.6
• Pr(b|A) ∝Pr(b)×Pr(A|b) = 0.8×(1-0.8)
• sincere voting: b

13

Sincere voting vs. informative voting

Ø Setting

• Two alternatives {a, b}, two signals {A,B}
• Three agents
• Pr(A|a) = pa=0.6, Pr(B|b)=pb=0.8
• Prior: Pr(a)=0.2, Pr(b)=0.8

Ø An agent receives A, other two agents are informative

• Conditioned on other two votes being {a, b}
• Signal profile is (A,A,B)
• Posterior probabilities
• Pr(a|A,A,B) ∝Pr(a)×Pr(A|a)×Pr(A|a)×Pr(B|a)=Pr(a)pa2(1-pa)

=0.2×0.62×(1-0.6)=0.0288

• Pr(b|A,A,B) ∝Pr(b)×Pr(A|b)×Pr(A|b)×Pr(B|b)=Pr(b) (1-pb)2pb

=0.8×(1-0.8)2×0.8=0.0256

• Strategic voting: a

14

Strategic voting

ØOutcome space O = {o1,…, om}

• Example: {Sunny, Rainy}

ØAn expert is asked for distribution q=(q1,…, qm) over O

• her true belief is p

ØSuppose the next day the weather is o, expert is awarded by a scoring rule s(q,o)

• s: Lot(O) × O à R
• Example: linear scoring rule slin(q,ok) = qk

ØExpert’s expected utility

• S(q,p) = ∑o∈Op(o)s(q,o)
• When p = (0.7, 0.3), S(q,p) = 0.7 q1 + 0.3 (1-q1), maximized

at q1=1

15

Eliciting Probabilities

Ø A scoring rule s is strictly proper, if for all p,q∈Lot(O) such that p≠q S(p,p) > S(q,p)

• reporting true belief is strictly optimal

Ø Example (logarithm scoring rule).

• slog(q,ok) = ln (qk)
• For k =2, Slog(q,p) = p1ln(q1) + (1-p1) ln(1-q1)
• maximized at q1 = p1

Ø Example (quadratic scoring rule).

• slog(q,ok) = 2qk - ∑jqj2
• For k =2, Slog(q,p) = 2p1q1 + 2(1-p1)(1-q1)-∑jqj2
• maximized at q1 = p1

16

(Strictly) Proper Scoring Rules

Ø Theorem. For m=2, a scoring rule s(q,p) is strictly proper, if and only if G(p) = S(p,p) is strict convex.

• Can be extended to m>2

17

Characterization of Strictly Proper Scoring Rules