Announcements Ø HW 3 due next Tuesday Ø No HW 4 1
CS6501: T opics in Learning and Game Theory (Fall 2019) Crowdsourcing Information and Peer Prediction Instructor: Haifeng Xu
Outline Ø Eliciting Information without Verification Ø Equilibrium Concept and Peer Prediction Mechanism Ø Bayesian Truth Serum 3
Crowdsourcing Information Ø Recruit AMT workers to label images • Cannot check ground truth (too costly) 4
Crowdsourcing Information Ø Recruit AMT workers to label images • Cannot check ground truth (too costly) Ø Peer grading (of, e.g., essays) on MOOC • Don’t know true scores 5
Crowdsourcing Information Ø Recruit AMT workers to label images • Cannot check ground truth (too costly) Ø Peer grading (of, e.g., essays) on MOOC • Don’t know true scores Ø Elicit ratings for various entities (e.g., on Yelp or Google) • We never find out the true quality/rating 6
Crowdsourcing Information Ø Recruit AMT workers to label images • Cannot check ground truth (too costly) Ø Peer grading (of, e.g., essays) on MOOC • Don’t know true scores Ø Elicit ratings for various entities (e.g., on Yelp or Google) • We never find out the true quality/rating Ø And many other applications… 7
Common Features in These Applications Ø We (the designer) elicit information from population Ø Cannot or too costly to know ground truth • The reason of using crowdsourcing info elicitation • Key difference from prediction markets Ø Agents/experts may misreport Challenge : cannot verify the report/prediction Solution : let multiple agents compete for the same task, and score them against each other (thus the name “peer prediction”) Where else did we see a similar idea? 8
A Simple and Concrete Example Ø Elicit Alice’s and Bob’s truthful rating 𝐵, 𝐶 about UVA dinning • 𝐵, 𝐶 ∈ {𝐼𝑗ℎ, 𝑀𝑝𝑥} • There is a common joint belief: 𝑄 𝐵, 𝐶 = [𝐼, 𝐼] = 0.5; 𝑄( 𝐵, 𝐶 = ) [𝐼, 𝑀] = 0.24 ; 𝑄 𝐵, 𝐶 = [𝑀, 𝐼] = 0.24 ; 𝑄 𝐵, 𝐶 = [𝑀, 𝑀] = 0.02 Let’s try to understand this distribution … Ø It is symmetric among Alice and Bob Ø 𝑄 𝐵 = 𝐼 = 0.5 + 0.24 = 0.74 Each expert very likely rates 𝐼 • >(?@A,B@A) C.D GD Ø 𝑄 𝐵 = 𝐼|𝐶 = 𝐼 = = C.EF = >(B@A) HE Given that one rates 𝐼 , the other very likely rates 𝐼 as well • Ø 𝑄 𝐵 = 𝐼|𝐶 = 𝑀 = >(?@A,B@I) = C.GF C.GJ = KG >(B@I) KH Given that one rates 𝑀 , the other still very likely rates 𝐼 • 9
A Simple and Concrete Example Ø Elicit Alice’s and Bob’s truthful rating 𝐵, 𝐶 about UVA dinning • 𝐵, 𝐶 ∈ {𝐼𝑗ℎ, 𝑀𝑝𝑥} • There is a common joint belief: 𝑄 𝐵, 𝐶 = [𝐼, 𝐼] = 0.5; 𝑄( 𝐵, 𝐶 = ) [𝐼, 𝑀] = 0.24 ; 𝑄 𝐵, 𝐶 = [𝑀, 𝐼] = 0.24 ; 𝑄 𝐵, 𝐶 = [𝑀, 𝑀] = 0.02 GD KG • 𝑄 𝐵 = 𝐼 = 0.74 ; 𝑄 𝐵 = 𝐼|𝐶 = 𝐼 = HE ; 𝑄 𝐵 = 𝐼|𝐶 = 𝑀 = KH Q : What are some natural peer comparison and rewarding mechanisms? Ø One simple idea is to reward agreement • Ask Alice and Bob to report their signals L 𝐵 , L 𝐶 (may misreport) • Award 1 to both if L 𝐵 = L 𝐶 , otherwise reward 0 10
A Simple and Concrete Example Ø Elicit Alice’s and Bob’s truthful rating 𝐵, 𝐶 about UVA dinning • 𝐵, 𝐶 ∈ {𝐼𝑗ℎ, 𝑀𝑝𝑥} • There is a common joint belief: 𝑄 𝐵, 𝐶 = [𝐼, 𝐼] = 0.5; 𝑄( 𝐵, 𝐶 = ) [𝐼, 𝑀] = 0.24 ; 𝑄 𝐵, 𝐶 = [𝑀, 𝐼] = 0.24 ; 𝑄 𝐵, 𝐶 = [𝑀, 𝑀] = 0.02 GD KG • 𝑄 𝐵 = 𝐼 = 0.74 ; 𝑄 𝐵 = 𝐼|𝐶 = 𝐼 = HE ; 𝑄 𝐵 = 𝐼|𝐶 = 𝑀 = KH Q : What are some natural peer comparison and rewarding mechanisms? Ø One simple idea is to reward agreement • Ask Alice and Bob to report their signals L 𝐵 , L 𝐶 (may misreport) • Award 1 to both if L 𝐵 = L 𝐶 , otherwise reward 0 Ø Does this work? • If 𝐵 = 𝐼 , what should Alice report? • If 𝐵 = 𝑀 , what should Alice report? 11
A Simple and Concrete Example Ø Elicit Alice’s and Bob’s truthful rating 𝐵, 𝐶 about UVA dinning • 𝐵, 𝐶 ∈ {𝐼𝑗ℎ, 𝑀𝑝𝑥} • There is a common joint belief: 𝑄 𝐵, 𝐶 = [𝐼, 𝐼] = 0.5; 𝑄( 𝐵, 𝐶 = ) [𝐼, 𝑀] = 0.24 ; 𝑄 𝐵, 𝐶 = [𝑀, 𝐼] = 0.24 ; 𝑄 𝐵, 𝐶 = [𝑀, 𝑀] = 0.02 GD KG • 𝑄 𝐵 = 𝐼 = 0.74 ; 𝑄 𝐵 = 𝐼|𝐶 = 𝐼 = HE ; 𝑄 𝐵 = 𝐼|𝐶 = 𝑀 = KH Q : What are some natural peer comparison and rewarding mechanisms? Ø One simple idea is to reward agreement • Ask Alice and Bob to report their signals L 𝐵 , L 𝐶 (may misreport) • Award 1 to both if L 𝐵 = L 𝐶 , otherwise reward 0 Ø Does this work? Truthful report is not an • If 𝐵 = 𝐼 , what should Alice report? equilibrium! • If 𝐵 = 𝑀 , what should Alice report? 12
A Simple and Concrete Example Ø Elicit Alice’s and Bob’s truthful rating 𝐵, 𝐶 about UVA dinning • 𝐵, 𝐶 ∈ {𝐼𝑗ℎ, 𝑀𝑝𝑥} • There is a common joint belief: 𝑄 𝐵, 𝐶 = [𝐼, 𝐼] = 0.5; 𝑄( 𝐵, 𝐶 = ) [𝐼, 𝑀] = 0.24 ; 𝑄 𝐵, 𝐶 = [𝑀, 𝐼] = 0.24 ; 𝑄 𝐵, 𝐶 = [𝑀, 𝑀] = 0.02 GD KG • 𝑄 𝐵 = 𝐼 = 0.74 ; 𝑄 𝐵 = 𝐼|𝐶 = 𝐼 = HE ; 𝑄 𝐵 = 𝐼|𝐶 = 𝑀 = KH Q : What are some natural peer comparison and rewarding mechanisms? Ø Both players always report 𝐼 (i.e., L 𝐵 = L 𝐶 = 𝐼 ) is a Nash Equ. Ø Why? • Well, under “rewarding agreement”, they both get 1 , the maximum possible • In fact, both always reporting 𝑀 is also a NE 13
Outline Ø Eliciting Information without Verification Ø Equilibrium Concept and Peer Prediction Mechanism Ø Bayesian Truth Serum 14
The Model of Peer Prediction Ø Two experts Alice and Bob, each holding a signal 𝐵 ∈ {𝐵 K , ⋯ , 𝐵 O } and 𝐶 ∈ {𝐶 K , ⋯ , 𝐶 P } respectively • A joint distribution 𝑞 of (𝐵, 𝐶) is publicly known • Everything we describe generalize to 𝑜 experts Ø We would like to elicit Alice’s and Bob’s true signals • We never know what signals they truly have A seemingly richer but equivalent model Ø We want to estimate distribution of random var 𝐹 Ø Joint prior distribution 𝑞 of (𝐵, 𝐶, 𝐹) is publicly known • E.g., 𝐹 is true quality of our dinning, which we never observe Ø Goal: elicit 𝐵, 𝐶 to refine our estimation of 𝐹 15
A Subtle Issue A seemingly richer but equivalent model Ø We want to estimate distribution of random var 𝐹 Ø Joint prior distribution 𝑞 of (𝐵, 𝐶, 𝐹) is publicly known • E.g., 𝐹 is true quality of our dinning, which we never observe Ø Goal: elicit 𝐵, 𝐶 to refine our estimation of 𝐹 Eliciting signals vs distributions Ø In prediction markets, we asked experts to report distributions Ø Here, could have done the same thing • Alice could report 𝑞 𝐹 𝐵 , the dist. of 𝐹 conditioned on her signal 𝐵 16
A Subtle Issue A seemingly richer but equivalent model Ø We want to estimate distribution of random var 𝐹 Ø Joint prior distribution 𝑞 of (𝐵, 𝐶, 𝐹) is publicly known • E.g., 𝐹 is true quality of our dinning, which we never observe Ø Goal: elicit 𝐵, 𝐶 to refine our estimation of 𝐹 Eliciting signals vs distributions Ø In prediction markets, we asked experts to report distributions Ø Here, could have done the same thing • Alice could report 𝑞 𝐹 𝐵 , the dist. of 𝐹 conditioned on her signal 𝐵 • Let’s make a minor assumption: 𝑞 𝐹 𝐵 ≠ 𝑞 𝐹 𝐵′ for any 𝐵 ≠ 𝐵′ • Then, reporting signal 𝐵 is equivalent to reporting distribution 𝑞 𝐹 𝐵 • So, w.l.o.g., eliciting signals is equivalent 17
A Subtle Issue A seemingly richer but equivalent model Ø We want to estimate distribution of random var 𝐹 Ø Joint prior distribution 𝑞 of (𝐵, 𝐶, 𝐹) is publicly known • E.g., 𝐹 is true quality of our dinning, which we never observe Ø Goal: elicit 𝐵, 𝐶 to refine our estimation of 𝐹 Eliciting signals vs distributions Ø In prediction markets, we asked experts to report distributions Ø Here, could have done the same thing • Alice could report 𝑞 𝐹 𝐵 , the dist. of 𝐹 conditioned on her signal 𝐵 • Let’s make a minor assumption: 𝑞 𝐹 𝐵 ≠ 𝑞 𝐹 𝐵′ for any 𝐵 ≠ 𝐵′ • Then, reporting signal 𝐵 is equivalent to reporting distribution 𝑞 𝐹 𝐵 • So, w.l.o.g., eliciting signals is equivalent Ø Drawback: have to assume an accurate and known prior 18
Info Elicitation Mechanisms and Equilibrium Ø Recall, we elicit info by asking Alice’s and Bob’s signal L 𝐵 , L 𝐶 ? (L 𝐵 , L B (L 𝐵 , L Ø As before, will design rewards 𝑠 𝐶 ) and 𝑠 𝐶 ) Ø Alice’s action is a report strategy 𝜏 ? 𝐵 ∈ {𝐵 K , ⋯ , 𝐵 O } [Bob similar] • This is a pure strategy • Will not consider mixed strategy here as we will design 𝑠 ? and 𝑠 B so that there is a good pure equilibrium • Truth-telling strategy: 𝜏 ? 𝐵 = 𝐵, 𝜏 B (𝐶) = 𝐶 Ø Then, what outcome is expected to occur? à equilibrium outcome Ø Generally, it is a Bayesian Nash equilibrium ( BNE ) • For simplicity, only define the equilibrium for our particular setting 19
Recommend
More recommend
