(Construction) Grammar does not Suffice for NLU Jerome Feldman, ICSI - - PowerPoint PPT Presentation

construction grammar does not suffice for nlu
SMART_READER_LITE
LIVE PREVIEW

(Construction) Grammar does not Suffice for NLU Jerome Feldman, ICSI - - PowerPoint PPT Presentation

(Construction) Grammar does not Suffice for NLU Jerome Feldman, ICSI & UC Berkeley Natural Language Understanding (NLU) is a manifest goal for applied linguistics, but its theoretical importance is not as obvious. Integrated form-meaning


slide-1
SLIDE 1

1

(Construction) Grammar does not Suffice for NLU

Jerome Feldman, ICSI & UC Berkeley

Natural Language Understanding (NLU) is a manifest goal for applied linguistics, but its theoretical importance is not as obvious. Integrated form-meaning pairs comprise the crux of Construction Grammar, but what is meaning? Decades of work at ICSI/UCB has established that embodied semantics (ECG) is necessary, but we also know that it is not sufficient. Language is inherently contextual and underspecified. An isolated grammar theory or program can only provide schematic analyses (SemSpecs in ECG) that are inadequate for full NLU. https://github.com/icsi-berkeley/ecg_homepage/wiki

slide-2
SLIDE 2

Natural Language Understanding

  • Natural Language Processing (NLP) is the overall category

– Search, Machine Translation, Sentiment Analysis, etc.

  • . Natural Language Understanding (NLU) ~ action without human intervention

– Google Search vs. Google Car

  • Why NLU might be of interest to ICCG

– Greater outside interest, support – CxG seems necessary for compositional approach – Full path understanding redefines many fundamental issues

  • Current Mainstream Approaches

– Templates: Siri, Cortana, Google, Alexa (next slide) – Machine Learning

  • Natural Language Generation adds more complications

– Habitability Problem – FCG – Luc Steels

Slide 2

slide-3
SLIDE 3

3

Amazon Alexa Skills ~ Templates

GetHoroscope what is the horoscope for { Sign} GetHoroscope what will be the horoscope for { Sign} be on {Date} GetHoroscope get me my horoscope … MatchSign do {FirstSign} and {Second Sign } get along MatchSign what is the relationship between {FirstSign} and {Second Sign }

developer.amazon.com/alexa-skills-kit

slide-4
SLIDE 4

The Winograd Challenge

http://www.cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html

All Pronoun Referent Resolution

  • 1. The city councilmen refused the demonstrators a permit because they

advocated/feared violence.

  • 3. The trophy doesn't fit into the brown suitcase because it is too large/small.
  • 5. Joan made sure to thank Susan for all the help she had received/given
  • 7. Paul tried to call George on the phone, but he wasn't successful/ available.
  • 9. The lawyer asked the witness a question, but he was reluctant to repeat/ answer

it.

  • 11. The delivery truck zoomed by the school bus because it was going so fast/slow.
  • 15. The man couldn't lift his son because he was so weak/heavy

4

slide-5
SLIDE 5

ECG2 - NLU Beyond the 1980s

1. Much more computation 2. NLP technology 3. Construction Grammar: form-meaning pairs Conceptual compositionality + Idioms, etc.

  • 4. Cognitive Linguistics: Conceptual primitives, Metaphor, etc.

ECG = Embodied Construction Grammar; 6 uses of formalism

  • 5. Constrained Best Fit : Analysis, Simulation, Learning

Analysis uses Bayesian (form, meaning and context) best fit

  • 6. Under-specification: Meaning involves context, goals, etc.

SemSpec = Semantic/Simulation Specification

  • 7. Simulation Semantics: Meaning as action/simulation
  • 8. CPRM= Coordinated Probabilistic Relational Models; Petri Nets ++

Action formalism works as a generative model

  • 9. Domain Semantics; Need rich semantics on the Action side

10. General NLU front end: Modest effort to link to a new Action side

Slide 5

slide-6
SLIDE 6

Language as Logic

Yet every sentence is not a proposition; only such are propositions that have in them truth or

  • falsity. Thus a prayer is a sentence, but it is

neither true nor false. Let us therefore dismiss all other types of sentences but the proposition, for this last concerns our present inquiry, whereas the investigation of others belongs rather to the study of rhetoric or poetry. Aristotle (De Interpretatione 17a1-8).

slide-7
SLIDE 7

Functionalism

In fact, the belief that neurophysiology is even relevant to the functioning of the mind is just a hypothesis. Who knows if we’re looking at the right aspects of the brain at all. Maybe there are other aspects of the brain that nobody has even dreamt of looking at yet. That’s often happened in the history of science. When people say that the mental is just the neurophysiological at a higher level, they’re being radically unscientific. We know a lot about the mental from a scientific point of view. We have explanatory theories that account for a lot of things. The belief that neurophysiology is implicated in these things could be true, but we have very little evidence for it. So, it’s just a kind of hope; look around and you see neurons: maybe they’re implicated. Noam Chomsky 1993, p.85

slide-8
SLIDE 8

Embodiment

Of all of these fields, the learning of languages would be the most impressive, since it is the most human of these

  • activities. This field, however, seems to

depend rather too much on the sense organs and locomotion to be feasible. Alan Turing (Intelligent Machines,1948)

< Continuity Principle of Darwin, American Pragmatists >

slide-9
SLIDE 9

Actionability in Integrated Cognitive Science

  • 1. All living things act; acting is what living things do.
  • 2. Natural selection constrains the fitness (utility) of these actions.
  • 3. Actionability is an agent's assessment of the expected utility of an external or

internal action.

  • 4. Volition is the key concept; agents perform volitional as well as automatic

actions

  • 5. This defines, but does claim to solve, actionability as a integrating issue for

Cognitive Science.

  • 6. No answers are suggested for hard mind-brain problems like subjective

agency.

  • 7. Actionability calculation often involves simulation of action and its

consequences.

Feldman JA(2016)Actionability and Simulation: No Representation without

  • Communication. Front.Psychol.7:1204. doi:10.3389/fpsyg.2016.01204

Slide 9

slide-10
SLIDE 10

Active representations

  • Many inferences about actions derive from what we know

about executing them

  • X-net representation based on stochastic Petri nets captures

dynamic, parameterized nature of actions

  • Used for acting, recognition, planning, and language

Walking: bound to a specific walker with a direction or goal consumes resources (e.g., energy) may have termination condition (e.g., walker at goal)

  • ngoing, iterative action

walker=Harry goal=home energy walker at goal

slide-11
SLIDE 11

“Indian government stumbling in implementing Liberalization Policy.”

slide-12
SLIDE 12

How do we specify an event? Formalized event schema

  • Key elements

– preconditions, resources, effects, sub-events – evoked by frames (alternatively: predicates, words)

  • Contrast with Event Recognition/Extraction, other NLP work

12

ISA hasFrame hasParameter

EVENT COMPOSITE EVENT

FRAME Actor Theme Instrument Patient CONSTRUAL Phase (enable, start, finish, ongoing, cancel) Manner (scales, rate, path) Zoom (expand, collapse) RELATION(E1,E2) Subevent Enable/Disable Suspend/Resume Abort/Terminate Cancel/Stop Mutually Exclusive Coordinate/Synch CONSTRUCT Sequence Concurrent/Conc. Sync Choose/Alternative Iterate/RepeatUntil(while) If-then-Else/Conditional PARAMETER Preconditions Effects Resources - In, Out Inputs Outputs Duration Grounding Time, Location

slide-13
SLIDE 13

Embodied Construction Grammar

  • Embodied representations

– active perceptual and motor schemas

(image schemas, x-schemas, frames, etc.)

– situational and discourse context

  • Construction Grammar

– Linguistic units pair form and meaning/function – Meaning pole based on (conceptual) Schemas. – Both constituency and (lexical) dependencies used.

  • Constraint-based

– Based on feature unification (as in LFG, HPSG) – Best fit: Diverse factors flexibly interact.

slide-14
SLIDE 14

Ideas from Cognitive Linguistics

  • Embodied Semantics (Lakoff, Langacker, Sweetser, Talmy)
  • Radial categories

(Rosch 1973, 1978; Lakoff 1985)

– mother: birth / adoptive / surrogate / genetic, …

  • Radical Construction Grammar (Croft 2001)

– Reference, Modification, (Predication -> Event Description)

  • Metaphor and metonymy (Lakoff & Johnson 1980, …)

– ARGUMENT IS WAR, MORE IS UP – The ham sandwich wants his check.

  • Mental spaces (Fauconnier 1994)

– The girl with blue eyes in the painting really has green eyes.

  • Conceptual blending (Fauconnier & Turner 2002, inter alia)

– workaholic, information highway, fake guns – “Does the name Pavlov ring a bell?”

slide-15
SLIDE 15

Conceptual Compositionality

Ontology Fragment (type motion sub process) (type create sub process cause) (type move sub motion) (type walk sub motion) (type enter sub motion) (type exit sub motion) (type run sub motion) (type drive sub motion)

slide-16
SLIDE 16

Semantic Specification

“the man built the house”

slide-17
SLIDE 17

ECG Workbench

ECG Workbench:

  • Based on Eclipse
  • Takes advantage of and fully integrates with

Eclipse RCP (Rich Client Platform)

  • Makes it easy to enter, edit and check consistency
  • f ECG grammars
  • Can analyze text licensed by the grammar,

producing a SemSpec (Semantic Specification)

  • Available at: https://github.com/icsi-berkeley/ecg_homepage
slide-18
SLIDE 18

18

slide-19
SLIDE 19

19

slide-20
SLIDE 20

20

slide-21
SLIDE 21

Embodied Construction Grammar: ECG

  • ECG serves:

1. as a technical tool for linguistic analysis 2. to specify shared grammar, conceptual conventions of a linguistic community 3. as a computer specification for implementing linguistic theories 4. as a representation for models and theories of language acquisition 5. as a front-end module for applied language-understanding tasks 6. as a high-level functional description for biological and behavioral experiments

slide-22
SLIDE 22

physics lowest energy state chemistry molecular fit biology fitness, MEU Neuroeconomics vision threats, friends language errors, NTL, OT Constrained Best Fit in Nature

inanimate animate

society, politics framing, compromise

slide-23
SLIDE 23
  • Mother (I) give you this (a

toy).

CHILDES Beijing Corpus (Tardiff, 1993; Tardiff, 1996)

ma1+ma gei3 ni3 zhei4+ge mother give 2PS this+CLS

  • You give auntie [the

peach].

  • Oh (go on)! You give [auntie]

[that].

Productive Argument Omission (Mandarin)

Johno Bryant & Eva Mok

1 2 3

ni3 gei3 yi2 2PS give auntie ao ni3 gei3 ya

EMP 2PS give EMP

4 gei3

give

  • [I] give [you] [some peach].
slide-24
SLIDE 24

Competition-based analyzer finds the best analysis

  • An analysis is made up of:

– A constructional tree – A set of contextual resolutions – A semantic specification

The best fit has the highest combined score

slide-25
SLIDE 25

Analyzing ni3 gei3 yi2 (You give auntie)

  • Syntactic Fit:

– P(Theme omitted | ditransitive cxn) = 0.65 – P(Recipient omitted | ditransitive cxn) = 0.42

Two of the competing analyses: ni3 gei3 yi2

  • mitted

↓ ↓ ↓ ↓

Giver Transfer Recipient Theme

ni3 gei3

  • mitted

yi2 ↓ ↓ ↓ ↓

Giver Transfer Recipient Theme

(1-0.78)*(1-0.42)*0.65 = 0.08 (1-0.78)*(1-0.65)*0.42 = 0.03

slide-26
SLIDE 26

26

Metaphors in ECG "he slid into poverty"

schema MetaphorSchema roles name: @metaphors source: @entity target: @entity schema stateAsLocationMetaphor subcase of MetaphorSchema roles name: @stateAsLocation source: @region target: @abstractState // reconstrues an RD's ontological-category to the source domain ~ best fit schema MetaphoricalRD subcase of RD roles met: MetaphorSchema general construction MetaphoricalNominal subcase of Nominal constructional constituents core: Nominal meaning: MetaphoricalRD evokes MetaphorSchema as met constraints self.m.number <--> core.m.number self.m.bounding <--> core.m.bounding self.m.scale <--> core.m.scale self.m.amount <--> core.m.amount self.m.givenness <--> core.m.givenness self.m.met <--> met

slide-27
SLIDE 27

Shared Ontology, Templates

Task API SemSpec USER Application (World) Problem Solver(s)

Speech/Text ECG Analyzer Specializer Observations

General NLU Framework Autonomous

Systems

N-tuples( ActSpec) : JSON agent communication

Commands and data App- interactions Situation model

N-tuples

Inferences

UI-Agent

slide-28
SLIDE 28

Robot1, move North then return!

slide-29
SLIDE 29

Robot1, move North then return!

Analyzer

slide-30
SLIDE 30

Robot1, move North then return!

slide-31
SLIDE 31

Parse serial processes Initialize executable template Assign values in SemSpec to template

Specializer

slide-32
SLIDE 32

Parameters of message First part of N-tuple received by Solver; contains information for first part of serial process (“move North”) Second part of N-tuple received by Solver; contains information for second part of serial process (“return”)

Struct(speed=0.5, action='move', protagonist='robot1_instance', distance=Struct(units='square', value=6), direction=None, control_state='ongoing', kind='execute', heading='north', goal=None)

N-Tuple ~ ActSpec

[Struct(protagonist='robot1_instance', heading='north', goal=None, kind='execute', speed=0.5, action='move', distance=Struct(units='square', value=8), direction=None, control_state='ongoing'), Struct(protagonist='robot1_instance', heading=None, goal={'location': 'home'}, kind='execute', speed=0.5, action='move', distance=Struct(units='square', value=8), direction=None, control_state='ongoing')] Struct(speed=0.5, action='move', protagonist='robot1_instance', distance=Struct(units='square', value=6), direction=None, control_state='ongoing', kind='execute', heading=None, goal={'location': 'home'})

slide-33
SLIDE 33

Solver

Unpacks N-tuple passed from Specializer. Determines heading of movement vector (“move North”). Returns to “home” position, stored in prior state variable.

slide-34
SLIDE 34

Morse (Robot Simulator)

Moves robot

Gets the new state of the world

Updates internal model of the world Gets the updated state of the world Moves the robot in simulation (and waits for it to arrive)

slide-35
SLIDE 35

35

User: Robot1, move to the box! Solver: Which box? User: The red one Solver: Which red one ? User: The one near the green box. Solver: Begin move to destination.

Clarification Dialog

slide-36
SLIDE 36

36

slide-37
SLIDE 37

37

ECG2 NLU Framework ~ again

slide-38
SLIDE 38

Modularity and Retargeting

  • Modularity of the system permits easy retargeting towards

new applications

  • Deltas: modifications that enable application-specific functionality.
  • Action:
  • Subclass core modules (e.g. Core Problem Solver)
  • Implement n-tuple communication for new modules
  • Language:
  • Add application-specific vocabulary (Token Tool)
  • Define application-specific N-tuple templates
  • Import or create application-specific schemas and

constructions (package system).

  • Product: what the end user sees/uses
  • Developer uses ECG 2.0 environment to generate a product
  • e.g., Robot Control, StarCraft Interface
slide-39
SLIDE 39

ECG 2.0 NLU Environment

  • Tools
  • ECG Workbench
  • Token Tool
  • Template Editor (JSON)
  • Package System, Git
  • Kaldi speech analysis
  • Celex morphology
  • Core modules
  • Core language
  • ECG Analyzer, Core Specializer, Core UI-Agent, Core ECG

Grammar

  • Core action
  • Core Problem Solver
  • Core Communication:
  • Transport, N-tuple templates, Core Agent
  • Core modules function as an integrated starter application.
slide-40
SLIDE 40

Software Tools: Token Tool

slide-41
SLIDE 41

Deltas for a New Application (Pt. 1)

Identify application domain and relevant vocabulary, and design API.

Domain: Economic Policy Metaphors Vocabulary:

  • Source: MOTION (stumble, collapse…),

HEALTH (cure, prescribe, sick…)

  • Target: ECONOMIC POLICY (economy,

depression…) App: after-KARMA (Narayanan 1997)

Design n-tuple templates to convey semantics to Problem Solver. Action_is_Motion: { mover: @India actionary: @stumble aspect: @progressive frame: SelfMotion action: { actionary: @implement createdThing: @policy creator: @India frame: Creation } }

slide-42
SLIDE 42

“Indian government stumbling in implementing Liberalization Policy.”

slide-43
SLIDE 43

Deltas for a New Application (Pt. 2)

  • Add

relevant tokens (and schemas/cxns, if necessary) to grammar.

  • Extend existing “Core”

Specializer and Problem Solver as needed for application

  • Build and test the new product.

Examples:

  • India is stumbling in

implementing the policy.

India :: Country :: self.m.referent <-- @India Stumble :: SelfMotion :: self.m.actionary <-- @stumble Policy :: AbstractNoun :: self.m.ontological-category <-- @poli Implement :: CreationType :: self.m.actionary <-- @implemen

TBD

slide-44
SLIDE 44

Problem ~ Indirect Speech Acts

  • Utterances (“it’s warm in here”) can have

three “levels” of meaning (Austin 1962):

— Locutionary: literal interpretation (e.g. assertion about

room temperature)

— Illocutionary: the intention (e.g. request to open a

window)

— Perlocutionary: the effect (e.g. the listener actually

  • pening the window)

— Indirect speech act: literal interpretation is distinct from

the intention of the utterance.

slide-45
SLIDE 45

Solution in Construction Grammar

— Illocutionary Force Indicating Devices (IFIDs): indirect

speech acts common in certain grammatical patterns (Searle & Vanderveken, 1985)

— Define constructions to exploit best-fit analysis of ECG

Analyzer — Could you push the blue box 5 inches south?

slide-46
SLIDE 46

mood mood

S = {“is the blue box big?”, “did Robot1 push the green box?” “could you push the blue box?”…} S = {“could you push the blue box?”…}

slide-47
SLIDE 47
  • Resulting n-tuple then has indirect in the

SpeechAct slot Solution In Construction Grammar

— System could request clarification if uncertain of indirect

speech act interpretation

SpeechAct: indirect Predicate_type: command Retury_type: error_descriptor Content: {….}

slide-48
SLIDE 48

mood

slide-49
SLIDE 49

mood

slide-50
SLIDE 50

Embodied Construction Grammar: ECG

  • ECG serves:

1. as a technical tool for linguistic analysis 2. to specify shared grammar, conceptual conventions of a linguistic community 3. as a computer specification for implementing linguistic theories 4. as a representation for models and theories of language acquisition 5. as a front-end system for applied language-understanding tasks 6. as a high-level functional description for biological and behavioral experiments

slide-51
SLIDE 51

51

Bryant,J. Best-Fit Constructional Analysis. PhD thesis, University of California at Berkeley, 2008. Eppe,M., S.Trott, V. Raghuram, J. Feldman and A. Janin “Application-Independent and Integration- Friendly Natural Language Understanding” Global Conference on Artificial Intelligence (GCAI 2016), EPiC Series in Computing, 2016 Feldman J. From Molecule to Metaphor: A Neural Theory of Language. MIT Pres, 2006 Feldman, J. , J. Bryant, and E. Dodge. A Neural Theory of Language and Embodied Construction

  • Grammar. In The Oxford Handbook of Computational Linguistics, pages 38 – 111. Oxford University

Press, 2009. Feldman, J. Embodied Language, Best-fit Analysis, and Formal Compositionality, Physics of Life Reviews, Vol. 7, #4, pp. 385-410, 2010 Feldman, J. (2016) Actionability and Simulation:NoRepresentation withoutCommunication. Front.Psychol.7:1204. doi:10.3389/fpsyg.2016.01204 Trott,S. M. Eppe and J. Feldman “Communicating Intentions in Human-Robot Interaction” International Symposium on Robot and Human Interactive Communication (RO-MAN 2016)

References

slide-52
SLIDE 52

52

slide-53
SLIDE 53

Computing with Natural Language

  • An integrated system combining

– Deep semantic analysis of language in context with – A scalable simulation model

  • Best-fit Language Analyzer

– Embodied Construction Grammar (ECG)

  • Construction Parser

– John Bryant PhD Thesis 2008

– Eva Mok PhD Thesis 2009 – Ellen Dodge PhD Thesis 2010

  • Scalable Domain Representation

– Event Models

– Steve Sinha PhD Thesis 2008 – Joe Makin PhD Thesis 2008

– Coordinated Probabilistic Relational Models

– Leon Barrett PhD Thesis 2010 – Steve Doubleday Thesis 2015

slide-54
SLIDE 54

Cognitive Science and Neuroscience

Science is a reductionist enterprise - we look for explanations of phenomena at more basic levels. This does not entail "eliminative reduction" where only the lowest level has explanatory power. Theory, modelling, and experiment at multiple levels is important and these should be consistent. For Cognitive Science, the ancient formulation of knowledge as truth may be a serious barrier to understanding the mapping of thought to neurobiology and beyond.

slide-55
SLIDE 55

Actionability, Simulation and Unified Cognitive Science

1) Action is evolutionarily much older than symbolic thought, belief, etc.; also developmentally earlier 2) Only living things act (in our sense); natural forces, mechanisms act by metaphorical extension. 3) Fitness is nature’s assessment of actions; we define actionability as an organism’s internal assessment of its available actions in context. 4) Actionability, not non-tautological truth, is what an agent/animal can actually compute. 5) Communication is action and is needed for cooperation – from pheromones to language 6) Actions include persistent change of internal state: self-concept, memory, world models, learning, etc. The external world (e.g., other agents) is not static - internal models need simulation 7) The brain is not a set of areas that represent things, rather a network of circuits that do things. 8) In animals, perception is best-fit, active, and utility/affordance based. 9) Mysteries remain; subjective experience, binding, self, free will, robots, etc. 10) One crucial divide/cline is volitional action and communication – boundary not clear, but birds are above the line; protozoans, plants below. Assume, in nature, neurons are necessary for volition. 11) Volitional actions have automatic components and influence, e.g., speech 12) Cognitive Science is bounded by [neurons, individuals]; unify with related sciences. 13) Overall goal of the effort is consistency with all experimental findings. 14) Theory remains central; multiple formalisms are needed – theories should cohere Control, probability, computation, logic, dynamics, utility, process, system, learning, 15) Formulation is multi-level in three ways: Standard divisions by scale, complexity - synapse, neuron, circuit, etc. System formulation – whole and parts inseparable, body-environment coupling essential Higher level sciences describe the phenomena, e.g., linguistics, psychology. 16) Action models are multi-modal: describe execution, recognition, planning, language. 17) Volitional simulation proposed as the mechanism of planning, mind-reading, etc. With an appropriate formalism, simulation can yield both causal and predictive inferences. 18) Biological, social, and cultural co-evolution, including language. 19) Linguistics based on embodied simulation semantics as the foundation of language and thought. 20) Additional mechanisms include construction grammar, mental spaces, mappings, etc. 21) Rationalization and other mental illusions

slide-56
SLIDE 56

Unified Cognitive Science

Neurobiology Psychology Computer Science Linguistics Philosophy Social Sciences Experience Take all the Findings and Constraints Seriously

slide-57
SLIDE 57

ECG Workbench

ECG Workbench:

  • Based on Eclipse
  • Takes advantage of and fully integrates with

Eclipse RCP (Rich Client Platform)

  • Makes it easy to enter, edit and check consistency
  • f ECG grammars
  • Can analyze text licensed by the grammar,

producing a SemSpec (Semantic Specification)

  • Download: http://www1.icsi.berkeley.edu/~lucag/
slide-58
SLIDE 58

Same Examples in Spanish

Simple Commands: Robot1, muévete al norte! Robot1, muévete detrás de una caja roja! Robot1, empuje la caja azul al norte! Serial Processes: Robot1, muévete a la caja azul y regresa! Robot1, muévete a la caja roja y grande y muévete a la caja roja y pequeña! Conditionals + Referent Resolution: Robot1, si la caja verde está cerca de la caja roja y pequeña, la empuje al norte! Robot1, si la caja pequeña es roja, muévete a la caja verde y la empuje! Questions: es la caja cerca de la caja azul verde? está la caja verde cerca de la caja roja y pequeña? cuáles cajas son rojas? cuáles cajas están cerca de la caja roja y pequeña? dónde está la caja verde?

slide-59
SLIDE 59

59

OBJECTIVES

Perform and integrate basic research and converging evidence from cognitive linguistics, computation, and neural information processing to build and exploit cognitive models of language understanding (NLU) and use. Provide and distribute an operational computational framework of action/simulation semantics to investigate the interaction between language, action, and cognition. Action/Simulation semantics offers the possibility to build systems for computing with natural language that come close to human performance levels. This is necessary for joint action in complex naval scenarios with a mix of human and artificial agents.

slide-60
SLIDE 60

60

OUTLINE also HANDOUT

1.

  • Intro. NLU is not NLP, good for CXG, not solved, approachable

2. Grammar does not suffice, CxG as {form, meaning} but what is meaning? 3. NTL/ECG started from meaning, brain, neuro-computation 4. Quote slides, then active meaning, Actionability, Simulation 5. Now ECG intro, workbench basics , examples 6. Best fit 7. Example analyses still just SemSpec. Maybe Spanish, 8. Metaphor 9. NLU, beyond the SemSpec, system diagram on hand-out

  • 10. Morse robot example
  • 11. Github release
  • 12. Compositionality, Intention example
slide-61
SLIDE 61

61

slide-62
SLIDE 62

Action Language Understanding System

  • Demonstrate utility through a series of scalable

prototypes

– that show the ability of the system to handle increasingly complex language – in a general way across multiple tasks and environments – to support communication in communities comprised of both human and artificial agents

  • Current Goal: Implement a prototype system that can

follow instructions and synthesize actions and procedures expressed in natural language.

– This requires the system to analyze natural language and translate this language in context into a coordinated network

  • f actions and complex commands.

Slide 62

slide-63
SLIDE 63

Software Tools: ECG Workbench

slide-64
SLIDE 64

Robot1, move North then return!

slide-65
SLIDE 65

65

slide-66
SLIDE 66

Shared Ontology, Templates

Task API SemSpec USER Application (World) Problem Solver(s)

Speech/Text ECG Analyzer Specializer Observations

General NLU Framework Autonomous

Systems

ECG: Embodied Construction Grammar N-tuples: JSON agent communication

Commands and data App- interactions Situation model

N-tuples

Inferences

UI-Agent

slide-67
SLIDE 67

— Discourse “moods”: assertion, query, command

Indirect Speech Acts

— Sometimes the literal “mood” is distinct from the

intended “mood” — Could you push the blue box 5 inches south?

— Solution: exploit ECG compositionality and best-fit

constraints to identify intended mood