Consensus Agreement Games Paul Upchurch, Daniel Sedra, Andrew - - PowerPoint PPT Presentation

▶
consensus agreement games
SMART_READER_LITE
LIVE PREVIEW

Consensus Agreement Games Paul Upchurch, Daniel Sedra, Andrew - - PowerPoint PPT Presentation

Feb 05, 2024 334 likes •775 views

Consensus Agreement Games Paul Upchurch, Daniel Sedra, Andrew Mullen, Haym Hirsh and Kavita Bala Cornell University HCOMP 2016 Lamp Plate Plate Glass Food Fabric Dining room Talking Eating OpenSurfaces, hard-to-label examples Ceramic

slide-1
SLIDE 1

Consensus Agreement Games

Paul Upchurch, Daniel Sedra, Andrew Mullen, Haym Hirsh and Kavita Bala Cornell University HCOMP 2016

slide-2
SLIDE 2
slide-3
SLIDE 3

Lamp Plate Plate

slide-4
SLIDE 4

Glass Food Fabric

slide-5
SLIDE 5

Dining room Talking Eating

slide-6
SLIDE 6
slide-7
SLIDE 7

Ceramic or glass?

OpenSurfaces, hard-to-label examples

slide-8
SLIDE 8

Ceramic or glass? Painted or wood?

OpenSurfaces, hard-to-label examples

slide-9
SLIDE 9

Ceramic or glass? Ceramic, plastic

  • r foliage?

Painted or wood?

OpenSurfaces, hard-to-label examples

slide-10
SLIDE 10

Ceramic or glass? Ceramic, plastic

  • r foliage?

Painted or wood? Plastic or glass?

OpenSurfaces, hard-to-label examples

slide-11
SLIDE 11

Worker Error Rate by Agreement

paint paint paint paint paint MTurk workers Experts

0% error

paint paint paint

slide-12
SLIDE 12

Worker Error Rate by Agreement

paint paint paint paint paint

wood

paint paint paint paint MTurk workers Experts

0% error 5% error

paint paint paint paint paint paint

slide-13
SLIDE 13

Worker Error Rate by Agreement

0% error 5% error 27% error

paint paint paint paint paint

wood

paint paint paint paint

wood wood

paint paint paint paint paint paint paint paint paint paint paint paint MTurk workers Experts

slide-14
SLIDE 14

Worker Error Rate by Agreement

0% error 5% error 27% error

paint paint paint paint paint

wood

paint paint paint paint

wood wood

paint paint paint paint paint paint paint paint paint paint paint paint MTurk workers Experts

slide-15
SLIDE 15

Get More Worker Opinions?

7 votes 19.4% error

slide-16
SLIDE 16

Get More Worker Opinions?

7 votes 19.4% error 11 votes 19.0% error

slide-17
SLIDE 17

Get More Worker Opinions?

7 votes 19.4% error 11 votes 19.0% error 21 votes 19.6% error

slide-18
SLIDE 18

Cost Error rate

Expert Labels MTurk 7 votes

Cost vs. Quality on 3 out of 5 images

slide-19
SLIDE 19

Cost vs. Quality

Expert Labels MTurk 7 votes MTurk 11 votes MTurk 21 votes

Cost Error rate

slide-20
SLIDE 20

Cost vs. Quality

Expert Labels MTurk 7 votes MTurk 11 votes MTurk 21 votes Consensus Agreement Games

Cost Error rate

slide-21
SLIDE 21

Outline

The problem Related Work Our method Experimental results

slide-22
SLIDE 22

Related Work

Image labeling games. The ESP Game. Luis von Ahn and Laura Dabbish. (2004)

slide-23
SLIDE 23

Related Work

Image labeling games. The ESP Game. Luis von Ahn and Laura Dabbish. (2004) Consensus on graphs. Network graph coloring. Stephen Judd, Michael Kearns and Yevgeniy

  • Vorobeychik. (2010)
slide-24
SLIDE 24

Related Work

Image labeling games. The ESP Game. Luis von Ahn and Laura Dabbish. (2004) Consensus on graphs. Network graph coloring. Stephen Judd, Michael Kearns and Yevgeniy

  • Vorobeychik. (2010)

Peer agreement methods. Peer Truth Serum. Boi Faltings, Pearl Pu, Bao Duy Tran and Radu Jurca. (2014)

slide-25
SLIDE 25

Outline

The problem Related work Our method Experimental results

slide-26
SLIDE 26

Idea

slide-27
SLIDE 27

Idea

paper glass ?

slide-28
SLIDE 28

Idea

paper glass ? paper glass ?

slide-29
SLIDE 29

Consensus Agreement Games

Clique 1 Workers see all clique 1 votes Clique 2 Workers see all clique 2 votes

slide-30
SLIDE 30

No communication between cliques Shared image

Consensus Agreement Games

Clique 1 Workers see all clique 1 votes Clique 2 Workers see all clique 2 votes

slide-31
SLIDE 31

Time remaining: 120 seconds Payments: 1/8 cent + ½ cent consensus + 1 ½ cents agreement

Worker Experience

slide-32
SLIDE 32

No communication between cliques Clique 1 Workers see all clique 1 votes Shared image

Consensus Agreement Games

glass glass glass Clique 2 Workers see all clique 2 votes glass glass glass

Case 1: consensus agreement

slide-33
SLIDE 33

No communication between cliques Clique 1 Workers see all clique 1 votes Shared image

Consensus Agreement Games

paper glass paper Clique 2 Workers see all clique 2 votes glass paper glass

Case 2: no consensus

slide-34
SLIDE 34

No communication between cliques Clique 1 Workers see all clique 1 votes Shared image

Consensus Agreement Games

paper paper paper Clique 2 Workers see all clique 2 votes glass paper glass

Case 3: single consensus

slide-35
SLIDE 35

No communication between cliques Clique 1 Workers see all clique 1 votes Shared image

Consensus Agreement Games

paper paper paper Clique 2 Workers see all clique 2 votes glass glass glass

Case 4: consensus disagreement

slide-36
SLIDE 36

Outline

The problem Related work Our method Experimental results

slide-37
SLIDE 37

Ceramic or glass? Ceramic, plastic

  • r foliage?

Painted or wood? Plastic or glass?

OpenSurfaces, material recognition

http://opensurfaces.cs.cornell.edu/publications/opensurfaces/

slide-38
SLIDE 38

Bedroom, hotel room or nursery? River, ocean

  • r coast?

http://places.csail.mit.edu/

Places, scene recognition

slide-39
SLIDE 39

Clique size?

versus

slide-40
SLIDE 40

MTurk 7 votes 3 cents, 19% Consensus Agreement Games 12 cents, 12%

OpenSurfaces, cliques of 2

37% drop Cost Error rate

slide-41
SLIDE 41

Consensus Agreement Games 15 cents, 15%

Places, cliques of 3

Cost Error rate

MTurk 7 votes 3 cents, 24%

38% drop

slide-42
SLIDE 42

Consensus Agreement Games

Agreement Interactive Game

Payments: 1/8 cent + ½ cent consensus + 1 ½ cents agreement

Consensus Peer Incentives

slide-43
SLIDE 43

Questions?

Thanks for listening!