Judgment Aggregation Ulle Endriss Institute for Logic, Language and - PowerPoint PPT Presentation

Judgment Aggregation AUC Logic Lecture 2016 Judgment Aggregation Ulle Endriss Institute for Logic, Language and Computation University of Amsterdam Ulle Endriss 1

Judgment Aggregation AUC Logic Lecture 2016 Example Suppose three robots are in charge of climate control for this building. They need to make judgments on p (the temperature is below 17 ◦ C), q (we should switch on the heating), and p → q . p → q p q Robot 1: Yes Yes Yes Robot 2: No Yes No Robot 3: Yes No No Ulle Endriss 2

Judgment Aggregation AUC Logic Lecture 2016 Outline This will be an introduction to the theory of judgment aggregation . • Reminder: what you need to know about logic to understand this • The paradox of judgment aggregation: a second example • Main question: Is there a reasonable method of aggregation? Ulle Endriss 3

Judgment Aggregation AUC Logic Lecture 2016 Reminder: Satisfiability A set of propositional formulas is called satisfiable , if it has a model: thus, if you can assign truth values to the propositional variables so that all the formulas in the set become true. Which of the following sets are satisfiable? • {¬ q, p → q, p } • { p ∨ q, ¬ p ∨ ¬ q, r } • { p, q, ¬ ( p ∧ q ) } Ulle Endriss 4

Judgment Aggregation AUC Logic Lecture 2016 Example Three judges decide whether you are guilty of a breach of contract. Legal doctrine stipulates that you are guilty if and only if it is the case that the agreement was binding ( p ) and has not been honoured ( ¬ q ). p ∧ ¬ q p q Judge 1: Yes No Yes Judge 2: Yes Yes No Judge 3: No No No Ulle Endriss 5

Judgment Aggregation AUC Logic Lecture 2016 The Paradox of Judgment Aggregation Here are again our two examples: p → q p ∧ ¬ q p q p q Robot 1: Yes Yes Yes Judge 1: Yes No Yes Robot 2: No Yes No Judge 2: Yes Yes No Robot 3: Yes No No Judge 3: No No No Why paradox ? Two explanations: • Premise-driven rule and conclusion-driven rule disagree • Majority rule produces judgment set that is not satisfiable Ulle Endriss 6

Judgment Aggregation AUC Logic Lecture 2016 Judgment Aggregation An agenda Φ is a set of propositional formulas (and their negations). Example: Φ = { p, ¬ p, p → q, ¬ ( p → q ) , q, ¬ q } A judgment set J for the agenda Φ is a subset of Φ . We call J : • complete if ϕ ∈ J or ¬ ϕ ∈ J for all formulas ϕ, ¬ ϕ ∈ Φ • satisfiable if J has a model Now n individual agents each express judgments on the formulas in Φ , producing a profile J = ( J 1 , . . . , J n ) of complete and satisfiable sets. Example: J = ( { p, p → q, q } , {¬ p, p → q, ¬ q } , { p, ¬ ( p → q ) , ¬ q } ) An aggregation rule F for an agenda Φ and a group of n agents is a function mapping every given profile of complete and satisfiable individual judgment sets to a single collective judgment set. Ulle Endriss 7

Judgment Aggregation AUC Logic Lecture 2016 Example: Majority Rule Suppose three agents express judgments on the formulas in the agenda Φ = { p, ¬ p, q, ¬ q, p ∨ q, ¬ ( p ∨ q ) } . For simplicity, we only show the positive formulas in our tables: p ∨ q p q Agent 1: Yes No Yes J 1 = { p, ¬ q, p ∨ q } Agent 2: Yes Yes Yes J 2 = { p, q, p ∨ q } Agent 3: No No No J 3 = {¬ p, ¬ q, ¬ ( p ∨ q ) } The (strict) majority rule F maj takes a (complete and satisfiable) profile and returns the set of formulas accepted by > n 2 agents. In our example: F maj ( J ) = { p, ¬ q, p ∨ q } [complete and satisfiable!] Ulle Endriss 8

Judgment Aggregation AUC Logic Lecture 2016 Other Rules Instead of using the majority rule , we could also use: • Premise-driven rule: use majority voting on literals and infer other formulas from the literals accepted. • Quota-based rules: e.g., accept a formula iff � 2 3 of the agents do There are many more options. How to choose? Ulle Endriss 9

Judgment Aggregation AUC Logic Lecture 2016 The Axiomatic Method What makes for a “good” aggregation rule F ? The following so-called axioms all express intuitively appealing properties: • Anonymity: Treat all individual agents symmetrically! • Neutrality: Treat all formulas symmetrically! • Independence: To decide whether to accept ϕ , you should only have to consider which individual agents accept ϕ ! Observe that the majority rule satisfies all of these axioms. (But so do various other rules! Can you think of some examples?) Ulle Endriss 10

Judgment Aggregation AUC Logic Lecture 2016 Impossibility Theorem We have seen that the majority rule does not preserve satisfiability . Is there another “reasonable” rule that does not have this problem? Surprisingly, no! (at least not for certain agendas) Theorem 1 (List and Pettit, 2002) No judgment aggregation rule for � 2 agents and an agenda Φ with { p, q, p ∧ q } ⊆ Φ that satisfies the axioms of anonymity, neutrality, and independence will always return a collective judgment set that is complete and satisfiable. Remark: Also true for other agendas (e.g., all those we saw today). C. List and P. Pettit. Aggregating Sets of Judgments: An Impossibility Result. Economics and Philosophy , 18(1):89–110, 2002. Ulle Endriss 11

Judgment Aggregation AUC Logic Lecture 2016 Proof First, understand the impact of our three axioms: • Independence: acceptance of ϕ only depends on who accepts ϕ . • Add anonymity: it only depends on how many agents accept ϕ . • Add neutrality: must use same acceptance criterion for all formulas. We now prove the theorem for odd n (it’s even easier for even n ). Let N J ϕ be the set of agents who accept formula ϕ in profile J . Consider a profile J where n − 1 agents accept p and q ; one accepts p 2 but not q ; one accepts q but not p ; and n − 3 accept neither p nor q . 2 n +1 That is: | N J p | = | N J q | = | N J ¬ ( p ∧ q ) | = 2 . Then: • Accepting all three formulas contradicts satisfiability. • But if we accept none, completeness forces us to accept their complements, which also contradicts satisfiability. So it is indeed impossible to satisfy all of our requirements. � Ulle Endriss 12

Judgment Aggregation AUC Logic Lecture 2016 Related Research at the ILLC • Developing useful aggregation rules that guarantee satisfiability and other good properties: e.g., representative-voter rules . • Analysing under what circumstances an agent or a group of agents can benefit from strategically misrepresenting their judgments. • Designing JA-inspired methods for crowdsourcing of linguistic judgments, to support research in computational linguistics. U. Endriss and U. Grandi. Binary Aggregation by Selection of the Most Represen- tative Voter. Proc. 28th AAAI Conference on Artificial Intelligence (AAAI-2014). S. Botan, A. Novaro, and U. Endriss. Group Manipulation in Judgment Aggrega- tion. Proc. 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS-2016). C. Qing, U. Endriss, R. Fern´ andez, and J. Kruger. Empirical Analysis of Aggrega- tion Methods for Collective Annotation. Proc. 25th International Conference on Computational Linguistics (COLING-2014). Ulle Endriss 13

Judgment Aggregation AUC Logic Lecture 2016 Last Slide This has been an introduction to judgment aggregation . We’ve seen: • Formal framework for aggregating views on complex matters • Applicable to many diverse settings (thus: important) • Modelling coherent judgments as satisfiable sets of formulas • Paradox: majority view of coherent judges may be incoherent • Thus: need to carefully analyse the problem ( axiomatic method ) • Impossibility: no “reasonable” rule can always be coherent • Active research topic at the ILLC Ulle Endriss 14