consensus agreement games
play

Consensus Agreement Games Paul Upchurch, Daniel Sedra, Andrew - PowerPoint PPT Presentation

Feb 05, 2024 •334 likes •773 views

Consensus Agreement Games Paul Upchurch, Daniel Sedra, Andrew Mullen, Haym Hirsh and Kavita Bala Cornell University HCOMP 2016 Lamp Plate Plate Glass Food Fabric Dining room Talking Eating OpenSurfaces, hard-to-label examples Ceramic

  1. Consensus Agreement Games Paul Upchurch, Daniel Sedra, Andrew Mullen, Haym Hirsh and Kavita Bala Cornell University HCOMP 2016

  3. Lamp Plate Plate

  4. Glass Food Fabric

  5. Dining room Talking Eating

  7. OpenSurfaces, hard-to-label examples Ceramic or glass?

  8. OpenSurfaces, hard-to-label examples Ceramic or Painted or glass? wood?

  9. OpenSurfaces, hard-to-label examples Ceramic or Painted or Ceramic, plastic glass? wood? or foliage?

  10. OpenSurfaces, hard-to-label examples Ceramic or Painted or Ceramic, plastic Plastic or glass? wood? or foliage? glass?

  11. Worker Error Rate by Agreement paint paint paint paint paint paint paint paint 0% error MTurk workers Experts

  12. Worker Error Rate by Agreement paint paint paint paint paint paint paint paint 0% error paint paint paint paint paint paint paint wood 5% error MTurk workers Experts

  13. Worker Error Rate by Agreement paint paint paint paint paint paint paint paint 0% error paint paint paint paint paint paint paint wood 5% error paint paint paint paint paint paint wood wood 27% error MTurk workers Experts

  14. Worker Error Rate by Agreement paint paint paint paint paint paint paint paint 0% error paint paint paint paint paint paint paint wood 5% error paint paint paint paint paint paint wood wood 27% error MTurk workers Experts

  15. Get More Worker Opinions? 7 votes 19.4% error

  16. Get More Worker Opinions? 7 votes 19.4% error 11 votes 19.0% error

  17. Get More Worker Opinions? 7 votes 19.4% error 11 votes 19.0% error 21 votes 19.6% error

  18. Cost vs. Quality on 3 out of 5 images MTurk 7 votes Error rate Expert Labels Cost

  19. Cost vs. Quality MTurk MTurk MTurk 7 votes 11 votes 21 votes Error rate Expert Labels Cost

  20. Cost vs. Quality MTurk MTurk MTurk 7 votes 11 votes 21 votes Error rate Consensus Agreement Games Expert Labels Cost

  21. Outline The problem Related Work Our method Experimental results

  22. Related Work Image labeling games. The ESP Game. Luis von Ahn and Laura Dabbish. (2004)

  23. Related Work Image labeling games. The ESP Game. Luis von Ahn and Laura Dabbish. (2004) Consensus on graphs. Network graph coloring. Stephen Judd, Michael Kearns and Yevgeniy Vorobeychik. (2010)

  24. Related Work Image labeling games. The ESP Game. Luis von Ahn and Laura Dabbish. (2004) Consensus on graphs. Network graph coloring. Stephen Judd, Michael Kearns and Yevgeniy Vorobeychik. (2010) Peer agreement methods. Peer Truth Serum. Boi Faltings, Pearl Pu, Bao Duy Tran and Radu Jurca. (2014)

  25. Outline The problem Related work Our method Experimental results

  26. Idea

  27. Idea paper glass ?

  28. Idea paper paper glass glass ? ?

  29. Consensus Agreement Games Clique 1 Clique 2 Workers see all clique 1 votes Workers see all clique 2 votes

  30. Consensus Agreement Games Clique 1 Clique 2 Workers see all clique 1 votes Workers see all clique 2 votes Shared image No communication between cliques

  31. Worker Experience Time remaining: 120 seconds Payments: 1/8 cent + ½ cent consensus + 1 ½ cents agreement

  32. Consensus Agreement Games Clique 1 Clique 2 Workers see all clique 1 votes Workers see all clique 2 votes Shared image glass glass glass glass glass glass No communication between cliques Case 1: consensus agreement

  33. Consensus Agreement Games Clique 1 Clique 2 Workers see all clique 1 votes Workers see all clique 2 votes Shared image paper glass glass paper paper glass No communication between cliques Case 2: no consensus

  34. Consensus Agreement Games Clique 1 Clique 2 Workers see all clique 1 votes Workers see all clique 2 votes Shared image paper glass paper paper paper glass No communication between cliques Case 3: single consensus

  35. Consensus Agreement Games Clique 1 Clique 2 Workers see all clique 1 votes Workers see all clique 2 votes Shared image paper glass paper paper glass glass No communication between cliques Case 4: consensus disagreement

  36. Outline The problem Related work Our method Experimental results

  37. OpenSurfaces, material recognition Ceramic or Painted or Ceramic, plastic Plastic or glass? wood? or foliage? glass? http://opensurfaces.cs.cornell.edu/publications/opensurfaces/

  38. Places, scene recognition Bedroom, hotel River, ocean room or nursery? or coast? http://places.csail.mit.edu/

  39. Clique size? versus

  40. OpenSurfaces, cliques of 2 MTurk 7 votes 3 cents, 19% 37% drop Error rate Consensus Agreement Games 12 cents, 12% Cost

  41. Places, cliques of 3 MTurk 7 votes 3 cents, 24% 38% drop Error rate Consensus Agreement Games 15 cents, 15% Cost

  42. Consensus Agreement Games Consensus Agreement Interactive Incentives Payments: Game Peer 1/8 cent + ½ cent consensus + 1 ½ cents agreement

  43. Questions? Thanks for listening!

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Lecture 21: Consensus - 1 1 Consensus is agreement. A question is seen as settled. Sometimes a

Lecture 21: Consensus - 1 1 Consensus is agreement. A question is seen as settled. Sometimes a

HPSC 1001/1901/2101/2901 WHAT IS THIS THING CALLED SCIENCE? Semester 2, 2020 Lecture 21: Consensus - 1 1 Consensus is agreement. A question is seen as settled. Sometimes a topic is no longer discussed, as the evidence is agreed to be

456 views • 33 slides
When Aeron Met Raft Martin Thompson - @mjpt777 What does Consensus mean? consensus

When Aeron Met Raft Martin Thompson - @mjpt777 What does Consensus mean? consensus

Cluster Consensus When Aeron Met Raft Martin Thompson - @mjpt777 What does Consensus mean? consensus noun \ k n- sen(t)-s s \ : general agreement : unanimity Source: http://www.merriam-webster.com/ consensus noun \ k

1.14k views • 68 slides
Number of rounds for Consensus Non-Uniform Consensus (Non-Uniform) Agreement: No two correct

Number of rounds for Consensus Non-Uniform Consensus (Non-Uniform) Agreement: No two correct

Number of rounds for Consensus Non-Uniform Consensus (Non-Uniform) Agreement: No two correct processes decide on different values Validity: If all processes start with the same value v V , then v is the only possible decision value

461 views • 20 slides
Chapter 14: Consensus and Agreement Ajay Kshemkalyani and Mukesh Singhal Distributed Computing:

Chapter 14: Consensus and Agreement Ajay Kshemkalyani and Mukesh Singhal Distributed Computing:

Chapter 14: Consensus and Agreement Ajay Kshemkalyani and Mukesh Singhal Distributed Computing: Principles, Algorithms, and Systems Cambridge University Press A. Kshemkalyani and M. Singhal (Distributed Computing) Consensus and Agreement CUP

905 views • 55 slides
Consensus and Dissent or: Meta - Consensus Consensus about what we have consensus

Consensus and Dissent or: Meta - Consensus Consensus about what we have consensus

Consensus and Dissent or: Meta - Consensus Consensus about what we have consensus on Building on Bitcoin Lisboa, Portugal -- 4 July 2018 Paul Sztorc Twitter: @truthcoin paul@tierion.com 16C8 1597 E76E 86E6 C01E F037 AA4B 3330

619 views • 37 slides
Hybrid Consensus: Efficient Consensus in the Permissionless Model Rafael Pass and Elaine Shi

Hybrid Consensus: Efficient Consensus in the Permissionless Model Rafael Pass and Elaine Shi

Hybrid Consensus: Efficient Consensus in the Permissionless Model Rafael Pass and Elaine Shi IC3 1 Blockchains Satoshi Nakamoto, 2009 Public database Shared & maintained between network participants Blockchain agreement protocol

296 views • 15 slides
23-04-18 Advanced Network Security 3. Agreement and consensus I: concepts and protocols for

23-04-18 Advanced Network Security 3. Agreement and consensus I: concepts and protocols for

23-04-18 Advanced Network Security 3. Agreement and consensus I: concepts and protocols for crash failures Jaap-Henk Hoepman Digital Security (DS) Radboud University Nijmegen, the Netherlands @xotoxot // * jhh@cs.ru.nl // 8 www.cs.ru.nl/~jhh

115 views • 8 slides
Towards Low-Latency Byzantine Agreement Protocols Using RDMA DSN Workshop on Byzantine Consensus

Towards Low-Latency Byzantine Agreement Protocols Using RDMA DSN Workshop on Byzantine Consensus

Towards Low-Latency Byzantine Agreement Protocols Using RDMA DSN Workshop on Byzantine Consensus and Resilient Blockchains Signe Rsch, Ines Messadi, Rdiger Kapitza, 2018-06-25 ruesch@ibr.cs.tu-bs.de Technische Universitt Braunschweig,

781 views • 47 slides
Reasoning about Consensus Protocols Ilya Sergey ilyasergey.net Consensus Common meaning :

Reasoning about Consensus Protocols Ilya Sergey ilyasergey.net Consensus Common meaning :

Reasoning about Consensus Protocols Ilya Sergey ilyasergey.net Consensus Common meaning : a way for a set of parties to come to a shared agreement. In computing : ensuring that among the values proposed by a collection of

767 views • 63 slides
Appeal of MIP6 Consensus Call Thomas Narten narten@us.ibm.com Misc Points Consensus was

Appeal of MIP6 Consensus Call Thomas Narten narten@us.ibm.com Misc Points Consensus was

Appeal of MIP6 Consensus Call Thomas Narten narten@us.ibm.com Misc Points Consensus was rough, would be nice if there was clearer agreement We don't formally vote, votes best used to get sense of community and to verify/find areas of

401 views • 4 slides
PARALLEL CONSENSUS PROTOCOL Joint work with Sajjad Rizvi and Srinivasan Keshav CONSENSUS PROBLEM

PARALLEL CONSENSUS PROTOCOL Joint work with Sajjad Rizvi and Srinivasan Keshav CONSENSUS PROBLEM

CANOPUS: A SCALABLE AND MASSIVELY Bernard Wong CoNEXT 2017 PARALLEL CONSENSUS PROTOCOL Joint work with Sajjad Rizvi and Srinivasan Keshav CONSENSUS PROBLEM Agreement between a set of nodes in the presence of failures Asynchronous

321 views • 30 slides
A Mean Field Games Approach to Consensus Problems Mojtaba Nourian McGill University, Montr

A Mean Field Games Approach to Consensus Problems Mojtaba Nourian McGill University, Montr

A Mean Field Games Approach to Consensus Problems Mojtaba Nourian McGill University, Montr eal, Canada Information and Control in Networks LCCC, Lunds University 23 October 2012 Joint work with Professors Peter Caines, Roland Malham e

557 views • 22 slides
Consensus Building Consensus is Consensus is finding an acceptable proposal that all members

Consensus Building Consensus is Consensus is finding an acceptable proposal that all members

Consensus Building Consensus is Consensus is finding an acceptable proposal that all members can support. Consensus is not A unanimous vote A majority vote Everyone 100% satisfied Consensus requires Time Active

351 views • 7 slides
Real-valued average consensus over noisy quantized channels Andrea Censi Richard Murray Control

Real-valued average consensus over noisy quantized channels Andrea Censi Richard Murray Control

Real-valued average consensus over noisy quantized channels Andrea Censi Richard Murray Control & Dynamical Systems, California Institute of Technology Consensus problems Consensus: reach the agreement of agent beliefs or agent states,

446 views • 16 slides
Distributed Algorithms (PhD course) Consensus SARDAR MUHAMMAD SULAMAN Consensus The

Distributed Algorithms (PhD course) Consensus SARDAR MUHAMMAD SULAMAN Consensus The

Distributed Algorithms (PhD course) Consensus SARDAR MUHAMMAD SULAMAN Consensus The processes use consensus to agree on a common value out of values they initially propose Reaching consensus is one of the most fundamental problems in

591 views • 38 slides
1 What makes consensus hard? Fischer, Lynch and Patterson Fundamentally, the issue revolves

1 What makes consensus hard? Fischer, Lynch and Patterson Fundamentally, the issue revolves

Consensus a classic problem Consensus, impossibility Consensus abstraction underlies many results and Paxos distributed systems and protocols N processes They start execution with inputs { 0,1} Ken Birman Asynchronous,

255 views • 8 slides
!"#$%$&'&()&*+,"%-.&*%'+/#01'(+234-5)+6789: 3'&;51+<"11(

!"#$%$&'&()&*+,"%-.&*%'+/#01'(+234-5)+6789: 3'&;51+<"11(

Cmput 651 - Clique Trees {20,24}/10/2008 !"#$%$&'&()&*+,"%-.&*%'+/#01'(+234-5)+6789: 3'&;51+<"11( /%)).1=+>"#=? @ABCADEF8BFABBG H1%0&?I:+J#''1"KL"&104%?+3.M+N 8

436 views • 42 slides
Multi-Clique-Width, a Powerful New Width Parameter Martin Frer Pennsylvania State University

Multi-Clique-Width, a Powerful New Width Parameter Martin Frer Pennsylvania State University

Multi-Clique-Width, a Powerful New Width Parameter Martin Frer Pennsylvania State University Why tree-width? Many combinatorial graph problems are NP- hard. Usually, they are easy for trees. One wants to extend feasibility to a

681 views • 21 slides
CSCI 3210: Computational Game Theory Computational Complexity Ref: Algorithm Design (Ch 8) on

CSCI 3210: Computational Game Theory Computational Complexity Ref: Algorithm Design (Ch 8) on

10/11/19 CSCI 3210: Computational Game Theory Computational Complexity Ref: Algorithm Design (Ch 8) on Blackboard Mohammad T . Irfan Email: mirfan@bowdoin.edu Web: www.bowdoin.edu/~mirfan Famous complexity classes u NP (non-deterministic

579 views • 18 slides
Probabilistic Graphical Models Lecture 8 Junction Trees CS/CNS/EE 155 Andreas Krause

Probabilistic Graphical Models Lecture 8 Junction Trees CS/CNS/EE 155 Andreas Krause

Probabilistic Graphical Models Lecture 8 Junction Trees CS/CNS/EE 155 Andreas Krause Announcements Homework 2 due next Wednesday (Nov 4) in class Start early!!! Project milestones due Monday (Nov 9) 4 pages of writeup, NIPS format

988 views • 34 slides
Deterministic subgraph detection in broadcast CONGEST Janne H. Korhonen Aalto University Joel

Deterministic subgraph detection in broadcast CONGEST Janne H. Korhonen Aalto University Joel

Deterministic subgraph detection in broadcast CONGEST Janne H. Korhonen Aalto University Joel Rybicki University of Helsinki 1. Introduction Introduction: CONGEST model CONGEST model n nodes, connected by communication links

642 views • 45 slides
Co-nondeterminism in compositions: A kernelization lower bound for a Ramsey-type problem Stefan

Co-nondeterminism in compositions: A kernelization lower bound for a Ramsey-type problem Stefan

Co-nondeterminism in compositions: A kernelization lower bound for a Ramsey-type problem Stefan Kratsch September 03, WorKer 2011, Vienna 1 Introduction Ramsey(k) Input: A graph G and an integer k . Parameter: k . Question: Does G contain an

395 views • 25 slides
CS 758/858: Algorithms http://www.cs.unh.edu/~ruml/cs758 Graph Problems Number Problem Wheeler

CS 758/858: Algorithms http://www.cs.unh.edu/~ruml/cs758 Graph Problems Number Problem Wheeler

CS 758/858: Algorithms http://www.cs.unh.edu/~ruml/cs758 Graph Problems Number Problem Wheeler Ruml (UNH) Class 22, CS 758 1 / 13 Graph Problems Reductions Clique Vertex Cover Break Number Problem Reductions to Graph

986 views • 14 slides
Conditional Random Fields Dietrich Klakow Overview Sequence Labeling Bayesian Networks

Conditional Random Fields Dietrich Klakow Overview Sequence Labeling Bayesian Networks

Conditional Random Fields Dietrich Klakow Overview Sequence Labeling Bayesian Networks Markov Random Fields Conditional Random Fields Software example Sequence Labeling Tasks Sequence: a sentence Pierre Vinken , 61

372 views • 34 slides

More recommend