Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Language as a system of conventions Convention (Lewis, 1969) Regularity R in the behavior of members of population P is a convention iff 1 almost everyone prefers to conform to R on condition that almost everyone else does; and 2 almost everyone would just as happily defect to alternative regularity R ′ if everyone else did. Smith et al. (2013) As a convention-based communication agent, I assume 1 there is a single set of linguistic conventions L 2 everyone knows L 3 everyone else believes that I know L 4 but (social anxiety!) I don’t really know L ! 5 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Plan for today 1 The Rational Speech Acts (RSA) model 2 Training effective literal listeners 3 The joint inferences of deeply pragmatic listeners 4 The Cards task-oriented dialogue corpus 5 Language and action together 6 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead The Rational Speech Acts model 1 The Rational Speech Acts (RSA) model 2 Training effective literal listeners 3 The joint inferences of deeply pragmatic listeners 4 The Cards task-oriented dialogue corpus 5 Language and action together Mike Frank Noah Goodman 7 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead The Rational Speech Acts model Definition (Literal listener) L 0 ( world | msg , L ) ∝ I ( world ∈ L ( msg )) P ( world ) |L ( msg ) | Definition (Pragmatic speaker) S 1 ( msg | world , L ) ∝ exp λ ( log L 0 ( world | msg , L ) − C ( msg )) Definition (Pragmatic listener) L 1 ( world | msg , L ) ∝ S 1 ( msg | world , L ) P ( world ) 8 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead The origins of RSA • Rosenberg & Cohen (1964): early Bayesian model of production and comprehension in reference games. • Lewis (1969): signaling systems (H. Clark 1996) • Rabin (1990): recursive strategic signaling • Camerer et al. (2004): cognitive hierarchy models for games of conflict and coordination • Franke (2008, 2009) and J¨ ager (2007, 2012): iterated best response • Golland et al. (2010): L 1 ( S 0 ) with semantic parsing • Frank & Goodman (2012): L 1 ( S 1 ( L 1 ( S 0 ))) 9 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead An ad hoc conversational implicature r 1 .5 r 2 .5 r 1 r 2 Referents Prior r 1 r 2 ‘glasses’ T T ‘glasses’ 0 ‘hat’ F T ‘hat’ 0 Messages Costs Figure: Scenario 10 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead An ad hoc conversational implicature r 1 r 2 ‘glasses’ . 75 . 25 ‘hat’ 0 1 L 1 r 1 .5 r 2 .5 r 1 r 2 ‘glasses’ ‘hat’ 1 0 r 1 Referents Prior . 33 . 67 r 2 S 1 r 1 r 2 ‘glasses’ T T ‘glasses’ 0 r 1 r 2 ‘hat’ F T ‘hat’ 0 ‘glasses’ . 5 . 5 ‘hat’ 0 1 Messages Costs L 0 Figure: Scenario Figure: Reasoning 10 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Experimental results • Implicatures encourage mutual exclusivity, a.k.a., the pidgeon-hole principle (E. Clark 1987; Frank et al. 2009). This reasoning is pervasive in communication. • Implicature reasoning in simple reference games is extremely well-supported (Vogel et al., 2014; Degen & Franke, 2012). • Eye-tracking studies have illuminated the time-course of implicature reasoning during sentence processing (Grodner & Sedivy, 2008; Huang & Snedeker, 2009; Grodner et al., 2010). • For first-language acquisition, simple reference games separate linguistic abilities from pragmatic abilities — and kids turn out to be pretty good at pragmatics (Stiller et al., 2011). 11 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead The role of context r 1 r 2 ‘glasses’ . 75 . 25 r 1 .5 ‘hat’ 0 1 r 2 .5 r 1 r 2 L 1 Referents Prior r 1 r 2 ‘glasses’ T T ‘glasses’ 0 ‘hat’ F T ‘hat’ 0 Messages Costs Figure: Scenario 12 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead The role of context r 1 r 2 ‘glasses’ . 5 . 5 r 1 .5 ‘hat’ 0 1 r 2 .5 r 1 r 2 L 1 Referents Prior 1 . 0 r 1 r 2 0 . 8 L 2 ( r 1 | glasses ) ‘glasses’ T T ‘glasses’ 0 0 . 6 ‘hat’ F T ‘hat’ 0 0 . 4 0 . 2 Messages Costs 0 . 0 0 1 2 3 4 5 Cost( hat ) Figure: Scenario 12 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead The role of context r 1 r 2 ‘glasses’ . 99 . 01 r 1 .5 ‘hat’ 0 1 r 2 .5 r 1 r 2 L 1 Referents Prior 1 . 0 r 1 r 2 0 . 8 L 2 ( r 1 | glasses ) ‘glasses’ T T ‘glasses’ 0 0 . 6 ‘hat’ F T ‘hat’ 0 0 . 4 0 . 2 Messages Costs 0 . 0 0 . 1 0 . 2 0 . 3 0 . 4 0 . 5 0 . 6 0 . 7 0 . 8 0 . 9 1 . 0 P ( r 1 ) Figure: Scenario 12 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead The role of context r 1 r 2 ‘glasses’ . 95 . 05 r 1 .5 ‘hat’ 0 1 r 2 .5 r 1 r 2 L 10 Referents Prior 1 . 0 r 1 r 2 0 . 8 L n ( r 1 | glasses ) ‘glasses’ T T ‘glasses’ 0 0 . 6 ‘hat’ F T ‘hat’ 0 0 . 4 0 . 2 Messages Costs 0 . 0 0 1 2 3 4 5 6 7 8 9 Depth of recursion Figure: Scenario 12 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Bounded rationality 1.00 1.00 Proportion responding Proportion responding 0.75 0.75 0.50 0.50 0.25 0.25 0.00 0.00 0 1 2 0 1 2 Inference Level Inference Level 1 0 0 1 0 0 “hat” “hat” 1 1 0 1 0 1 “glasses” “glasses” 0 1 1 “mustache” (Vogel et al., 2014) 13 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Bounded rationality 13 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Self-trained discriminative RSA r 1 r 2 Weights: ( 1 , 0 ) ‘glasses’ T T ‘hat’ F T (Vogel et al., 2014) 14 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Self-trained discriminative RSA r 1 r 2 Weights: ( 1 , 0 ) ‘glasses’ T T ‘hat’ F T r 1 r 2 r 3 ‘glasses’ T F F Weights: . . . ‘hat’ T F T ‘mustache’ F T T (Vogel et al., 2014) 14 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Self-trained discriminative RSA S elf T rain ( Games G ) 1 Initialize S = S 0 2 Repeat: 3 L = T rain L istener ( G , S ) # Train on S’s production prefs. 4 S = T rain S peaker ( G , L ) # Train on L’s construal prefs. 5 Return ( S , L ) (Vogel et al., 2014) 14 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Self-trained discriminative RSA Discriminative Best Response Recursive Bayesian Models • Learn to reason pragmatically using supervised learning Agents recursively reason about their • Map directly from contextual features to speaker intent interlocutor’s ¡ communicative behavior • Iteratively build training sets for speaker and listener • • glasses • • • • (Vogel et al., 2014) 14 / 49 • • •
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Self-trained discriminative RSA “Complex” Context ANN Accuracy on the Complex Condition 1.0 0.8 Listener Accuracy 0.6 0.4 Level 0 0.2 Level 1 Level 2 0.0 0 2 4 6 8 10 Training Iterations 0 1 2 Inference Level (Vogel et al., 2014) 14 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead 1 The Rational Speech Acts (RSA) model 2 Training effective literal listeners 3 The joint inferences of deeply pragmatic listeners 4 The Cards task-oriented dialogue corpus 5 Language and action together Angel Chang Will Monroe Sam Bowman Chris Manning 15 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Scene generation Show me an original 3d scene of a home office . . . 16 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Scene generation Show me an original 3d scene of a home office . . . 16 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Scene as denotations What's in a 3D scene { 'modelID': '7bdc0aac', 'position': [118.545639, 97.979499, 3.098599], 'scale': 0.087807, 'rotation': -1.088704 } 17 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Scene as denotations { 'modelID': '7bdc0aac', 'position': [118.545639, 97.979499, 3.098599], Field Value 'scale': 0.087807, name ellington armchair 'rotation': -1.088704 } id 7bdc0aac tags armchair, chair, ellington, haughton, sam, seating, woodmark category Chair wnlemmas armchair unit 0.028974 up [0, 0, 1] front [0, -1, 0] 17 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Scene generation corpus The room has three windows on one wall. There is a red bed in the back of the room. Along side the bed is a side chair that is red and white. This room has a bed with red bedding against the wall. Next to the bed is a chair. there is a antique looking bed with red covers and pillows in a room. next to it is a recliner chair with red padding. also there are windows. There is a there is a bed with five pillows on it, and next to it is a chair bed and There is a bed in the room with two pillows there is a and a small chair near to the right side of it. chair next There is a large grey bed in the bottom right corner to the bed. of the room. Above the bed is a small black chair. Floor to ceiling windows on back wall. Green bed with two pillows and black blanket. Lights recessed into right side wall. Light wood flooring. A chair is in the upper right hand corner There is a bed on the side of the room. There is a chair in the corner, next to the windows. I see a bed and a chair. 18 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Scene generation as semantic interpretation There is a 3 person couch and table in the center of the room. THE GOOD 19 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Scene generation as semantic interpretation An L shaped couch with a vase on the corner. THE BAD 19 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Scene generation as semantic interpretation Generated scenes It is a square-shaped room with a wooden floor covered by a tan rug and an intricate wallpaper. There is a tall window in the corner with a small ceiling and desk-type object. In the middle of the room there is a gray-and-black carefully furnished bed with a simplistic gray cupboard and lamp on the opposite side of it in relation to the corner window. THE UGLY 19 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Recursive neural networks for natural logic entails : 0 . 8 equals : 0 . 1 Softmax classifier contradicts : 0 . 05 independent : 0 . 05 Comparison N(T)N layer all reptiles walk vs. some turtles move all reptiles walk some turtles move Composition RN(T)N layers all reptiles move walk some turtles reptiles some all turtles Pre-trained or randomly initialized learned word vectors 20 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Experiments Simulated data • Learning the natural logic relational algebra � • Learning propositional logic theorem provers � • Learning to reason with quantifiers and negation � (Bowman et al., 2014a,b) 21 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Experiments Simulated data • Learning the natural logic relational algebra � • Learning propositional logic theorem provers � • Learning to reason with quantifiers and negation � Naturalistic data • WordNet relations 95% test training on 33% of the data • The SICK textual entailment challenge 76.9% test (Bowman et al., 2014a,b) 21 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead A new natural language inference corpus To date: entailment, contradiction, and independence sentences for 15.5k ImageFlickr pictures/captions. Image caption Entailment Contradiction Independent Three people with People have Three people Men and women political signs. signs displaying have signs are holding up political themes. promoting their political placards football team. at a rally. A person working A city employee is The town sheriff A woman who for the city begins working outdoors. is sitting on a tree works for the city cutting down a swing. is using a tree. chainsaw. 22 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead 1 The Rational Speech Acts (RSA) model 2 Training effective literal listeners 3 The joint inferences of deeply pragmatic listeners 4 The Cards task-oriented dialogue corpus 5 Language and action together 23 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Lexical uncertainty (Bergen et al., 2012, 2014; Potts et al., 2015) 24 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Lexical uncertainty 1 It’s a sofa, not a couch. (Bergen et al., 2012, 2014; Potts et al., 2015) 24 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Lexical uncertainty 1 It’s a sofa, not a couch. 2 synagogues and other churches (Bergen et al., 2012, 2014; Potts et al., 2015) 24 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Lexical uncertainty 1 It’s a sofa, not a couch. 2 synagogues and other churches 3 superb but not outstanding (Bergen et al., 2012, 2014; Potts et al., 2015) 24 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Lexical uncertainty 1 It’s a sofa, not a couch. 2 synagogues and other churches 3 superb but not outstanding 4 L ( world , L | msg ) ∝ P ( world ) P ( L ) S 1 ( msg | world , L ) (Bergen et al., 2012, 2014; Potts et al., 2015) 24 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Lexical uncertainty 1 It’s a sofa, not a couch. 2 synagogues and other churches 3 superb but not outstanding 4 L ( world , L | msg ) ∝ P ( world ) P ( L ) S 1 ( msg | world , L ) 5 L ( world | msg ) ∝ P ( world ) � P ( L ) S 1 ( msg | world , L ) L∈ L (Bergen et al., 2012, 2014; Potts et al., 2015) 24 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Anxious experts (Levy & Potts, 2015) 25 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Anxious experts 1 oenophile means wine lover (Levy & Potts, 2015) 25 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Anxious experts 1 oenophile means wine lover 2 the bow lute, such as the Bambara ndang, (Hearst, 1992) (Levy & Potts, 2015) 25 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Anxious experts 1 oenophile means wine lover 2 the bow lute, such as the Bambara ndang, (Hearst, 1992) 3 wine lover or oenophile (Levy & Potts, 2015) 25 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Anxious experts 1 oenophile means wine lover 2 the bow lute, such as the Bambara ndang, (Hearst, 1992) 3 wine lover or oenophile 4 synagogues and other churches (Levy & Potts, 2015) 25 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Anxious experts 1 oenophile means wine lover 2 the bow lute, such as the Bambara ndang, (Hearst, 1992) 3 wine lover or oenophile 4 synagogues and other churches 5 synagogues or churches (Levy & Potts, 2015) 25 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Anxious experts 1 oenophile means wine lover 2 the bow lute, such as the Bambara ndang, (Hearst, 1992) 3 wine lover or oenophile 4 synagogues and other churches 5 synagogues or churches 6 S 2 ( msg | world , L ) ∝ exp ( α log ( L 1 ( world | msg , L )) − β log ( L 1 ( L | msg )) − C ( msg )) (Levy & Potts, 2015) 25 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Contextual uncertainty 26 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Contextual uncertainty 1 Chris has to miss class today. 26 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Contextual uncertainty 1 Chris has to miss class today. 2 A friend tweeting about bread-baking and soccer: “Who could have predicted that?!” 26 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Contextual uncertainty 1 Chris has to miss class today. 2 A friend tweeting about bread-baking and soccer: “Who could have predicted that?!” 3 Hand me the fork. 26 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Contextual uncertainty 1 Chris has to miss class today. 2 A friend tweeting about bread-baking and soccer: “Who could have predicted that?!” 3 Hand me the fork. 4 L ( world , context | msg , L ) ∝ P ( context ) S 1 ( msg , | world , context , L ) P context ( world ) 26 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Joint emotional and informational goals 1 Hyperbole a. I told you a thousand times already. b. It took a million years to get the waiter to our table. c. The watch cost $5000. 2 Sarcasm a. Oh, that’s wonderful! (it’s terrible) b. Yeah, delicious. (disgusting) c. Sounds great. (sounds terrible) 3 Metaphor a. Juliet is the sun. b. I feel sick as a dog. c. Our new boss is a shark. (Kao et al., 2014a,b) 27 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead The Cards corpus 1 The Rational Speech Acts (RSA) model 2 Training effective literal listeners 3 The joint inferences of deeply pragmatic listeners 4 The Cards task-oriented dialogue corpus 5 Language and action together 28 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead The Cards world TYPE HERE Task description: Six Yellow boxes mark cards You are on 2D consecutive cards of in your line of sight. the same suit Move with the arrow keys or The cards you are holding these buttons. 29 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead The Cards world Gather six consecutive cards of a particular suit (decide which suit together), or determine that this is impossible. Each of you can hold only three cards at a time, so you’ll have to coordinate your efforts. You can talk all you want, but you can make only a limited number of moves. 29 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead The Cards world Gather six consecutive cards of a particular suit (decide which suit together), or determine that this is impossible. Each of you can hold only three cards at a time, so you’ll have to coordinate your efforts. You can talk all you want, but you can make only a limited number of moves. What’s going on? ⇓ Which suit should we pursue? ⇓ Which sequence should we pursue? ⇓ Where is card X ? 29 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead By the numbers • 1,266 transcripts • Game length mean: 373.21 actions (median 305, sd 215.20) • Actions: ◮ Card pickup: 19,157 ◮ Card drop: 12,325 ◮ Move: 371,811 ◮ Utterance: 45,805 ◮ Utt. length mean: 5.69 words (median 5, sd 4.74) ◮ Total word count: 260,788 ◮ Total vocabulary: ≈ 4,000 30 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Task-oriented dialogue corpora Corpus Task type Domain Task-orient. Docs. Format Switchboard discussion open very loose 2,400 aud/txt SCARE search 3d world tight 15 aud/vid/txt TRAINS routes map tight 120 aud/txt Map Task routes map tight 128 aud/vid/txt Columbia Games games maps tight 12 aud/txt Cards search 2d grid tight 1,266 txt in context Chief selling points for Cards: • Pretty large. • Controlled enough that similar things happen often. • Very highly structured — the only corpus whose release version allows the user to replay all games with perfect fidelity. 31 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Grounded semantics (literal listeners) “in the bottom you see the “in the top right of the opening on the bottom row” middle part of the board” “i’m in the center” ⇓ ⇓ ⇓ BOARD(entrance & bottom); H : 5.48 middle(top & right); H : 5.27 BOARD(middle); H : 7.37 Utterances as bags of words. No preprocessing for spelling correction, lemmatization, etc. Assign semantic tags using log-linear classifiers trained on the corpus data. 32 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Language and action, language as action 1 The Rational Speech Acts (RSA) model 2 Training effective literal listeners 3 The joint inferences of deeply pragmatic listeners 4 The Cards task-oriented dialogue corpus 5 Language and action together Adam Vogel Max Bodoia Dan Jurafsky 33 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Simplified Cards scenario Both players must find the ace of spades. [DialogBot home movie] 34 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Agent framework We want our agent to: • Make moves that are likely to lead it to the card. • Change its behavior based on observations it receives. • Respond to advice from the other player. • Give advice to the other player. Modeling the problem as a POMDP allows us to train agents that have these properties. (Vogel et al., 2013a,b) 35 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Approximate solutions take us only part of the way • Even approximate solutions tractable only for < 10K states. 36 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Approximate solutions take us only part of the way • Even approximate solutions tractable only for < 10K states. • Card loc. Agent loc. Partner loc. Partner’s card beliefs 231 × 231 × 231 × 231 ≈ 50K ≈ 12M ≈ 3B 36 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Approximate solutions take us only part of the way • Even approximate solutions tractable only for < 10K states. • Card loc. Agent loc. Partner loc. Partner’s card beliefs 231 × 231 × 231 × 231 ≈ 50K ≈ 12M ≈ 3B • Language as a representation for planning: 36 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Approximate solutions take us only part of the way • Even approximate solutions tractable only for < 10K states. • Card loc. Agent loc. Partner loc. Partner’s card beliefs 231 × 231 × 231 × 231 ≈ 50K ≈ 12M ≈ 3B • Language as a representation for planning: 36 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Belief-state approximation ¯ b t o 1 o 2 o 1 ¯ b t o 2 o 1 o 2 b o 1 b o 2 ¯ ¯ t + 1 t + 1 ¯ b t + 1 o 1 o 2 o 1 o 2 o 1 o 1 o 2 o 2 b o 1 , o 1 ¯ b o 1 , o 2 ¯ b o 2 , o 1 ¯ b o 2 , o 2 ¯ t + 2 t + 2 t + 2 t + 2 ¯ b t + 2 (a) Exact multi-agent belief tracking (b) Approximate multi-agent belief tracking 37 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead ListenerBot • A POMDP agent that learns to navigate its world and interpret language. • Driven by its small negative reward for not having the card and its large positive reward for finding it. • No sensitivity to the other player. • Literal listeners: each message msg denotes P ( world | msg ) • Bayes rule to incorporate these as observations. 38 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead ListenerBot example [ListenerBot home movie] 39 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead ListenerBot example [ListenerBot home movie] 39 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead ListenerBot example [ListenerBot home movie] 39 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead ListenerBot example “it’s on the left side” ⇓ board ( left ) ⇓ [ListenerBot home movie] 39 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead ListenerBot example “it’s on the left side” ⇓ board ( left ) ⇓ [ListenerBot home movie] 39 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead DialogBot DialogBot is a strict extension of Listener Bot: • The set of states is now all combinations of ◮ both players’ positions ◮ the card’s region ◮ the region the other player believes the card to be in • The set of actions now includes dialogue actions. • (The player assumes that) a dialogue action U alters the other player’s beliefs in the same way that U would impact his own. • Same basic reward structure as for Listenerbot, except now also sensitive to whether the other player has found the card. • Approximate RSA is a special case. 40 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead How the agents relate to each other o 0 o 2 a 2 s 0 s ¯ ¯ 2 s s 0 s s 0 s s 0 R R R o 0 o a o 0 o 1 a 1 o a o 0 1 (a) ListenerBot POMDP (b) Full Dec-POMDP (c) DialogBot POMDP 41 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead DialogBot and ListenerBot play together DialogBot beliefs ListenerBot beliefs DialogBot beliefs: DialogBot beliefs: ListenerBot’s position ListenerBot’s beliefs 42 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead DialogBot and ListenerBot play together DialogBot beliefs ListenerBot beliefs DialogBot beliefs: DialogBot beliefs: ListenerBot’s position ListenerBot’s beliefs 42 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead DialogBot and ListenerBot play together Dialogbot: “Top” DialogBot beliefs ListenerBot beliefs DialogBot beliefs: DialogBot beliefs: ListenerBot’s position ListenerBot’s beliefs 42 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead DialogBot and ListenerBot play together Dialogbot: “Top” DialogBot beliefs ListenerBot beliefs DialogBot beliefs: DialogBot beliefs: ListenerBot’s position ListenerBot’s beliefs 42 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead DialogBot and ListenerBot play together DialogBot beliefs ListenerBot beliefs DialogBot beliefs: DialogBot beliefs: ListenerBot’s position ListenerBot’s beliefs 42 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead DialogBot and ListenerBot play together DialogBot beliefs ListenerBot beliefs DialogBot beliefs: DialogBot beliefs: ListenerBot’s position ListenerBot’s beliefs 42 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead DialogBot and ListenerBot play together DialogBot beliefs ListenerBot beliefs DialogBot beliefs: DialogBot beliefs: ListenerBot’s position ListenerBot’s beliefs 42 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Emergent pragmatics Quality • The Gricean maxim of quality says roughly “Be truthful”. • For DialogBot, this emerges from the decision problem: false information is (typically) more costly. • DialogBot would lie if he thought it would move them toward the objective. Quantity and Relevance • The Gricean maxims of quantity and relevance for informative, timely contributions. • When DialogBot finds the card, he communicates the information, not because he is hard-coded to do so, but rather because it will help the other player find it. 43 / 49
Overview The RSA model Literal listeners Pragmatic listeners The Cards corpus Language and action Looking ahead Grown-up DialogBots (a week of policy exploration) 44 / 49
Recommend
More recommend
