MULTIMODAL SEMANTIC SIMULATIONS OF LINGUISTICALLY UNDERSPECIFIED - - PowerPoint PPT Presentation

▶
multimodal semantic simulations of linguistically
SMART_READER_LITE
LIVE PREVIEW

MULTIMODAL SEMANTIC SIMULATIONS OF LINGUISTICALLY UNDERSPECIFIED - - PowerPoint PPT Presentation

Oct 05, 2023 337 likes •591 views

Spatial Cognition 2016 MULTIMODAL SEMANTIC SIMULATIONS OF LINGUISTICALLY UNDERSPECIFIED MOTION EVENTS Nikhil Krishnaswamy and James Pustejovsky, Brandeis University August 5, 2016, Spatial Cognition 2016, Philadelphia, PA, USA Remarkable

slide-1
SLIDE 1

Spatial Cognition 2016

MULTIMODAL SEMANTIC SIMULATIONS OF LINGUISTICALLY UNDERSPECIFIED MOTION EVENTS

Nikhil Krishnaswamy and James Pustejovsky, Brandeis University August 5, 2016, Spatial Cognition 2016, Philadelphia, PA, USA

slide-2
SLIDE 2

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Foundations

  • Remarkable number of concepts in human mental model
  • Mental models are adaptable
  • Can make sense of new situations, contexts, and

kinds of knowledge

  • Can be revised based on new experience
  • Mental models are embodied and multimodal
  • Embodiment maps concepts between domains
  • Modalities (perceptual and effector) constitute

aspects of representation

  • “Simulation”: mental instantiation of an utterance, based
  • n embodiment

August 5, 2016 Philadelphia, PA, USA

2

slide-3
SLIDE 3

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Past/Related Research

  • Spatial/temporal algebraic interval logic
  • Allen Temporal Relations (Allen, 1983)
  • Region Connection Calculus (RCC8) (Randell et al., 1992)
  • RCC-3D (Albath, et al., 2010)
  • Generative Lexicon, DITL (Pustejovsky, 1995; Pustejovsky and

Moszkowicz, 2011)

  • Static scene generation
  • WordsEye (Coyne and Sproat, 2001)
  • LEONARD (Siskind, 2001)
  • Stanford NLP Group (Chang et al., 2015)
  • QSR/Game AI approaches to scenario-based simulation (Forbus et al.,

2001; Dill, 2011)

  • Spatial constraint mapping to animation (Bindiganavale and Badler, 1998)

August 5, 2016 Philadelphia, PA, USA

3

slide-4
SLIDE 4

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Allen T emporal Relations

August 5, 2016 Philadelphia, PA, USA

4

slide-5
SLIDE 5

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Region Connection Calculus

August 5, 2016 Philadelphia, PA, USA

5

slide-6
SLIDE 6

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

WordsEye

August 5, 2016 Philadelphia, PA, USA

6

slide-7
SLIDE 7

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Cognitive Linguistic Simulation

August 5, 2016 Philadelphia, PA, USA

7

“Enterp the parking lot” Path depends on bounds of parking lot “Enter” is a path verb (Pustejovsky and Moszkowicz, 2011)

slide-8
SLIDE 8

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Cognitive Linguistic Simulation

August 5, 2016 Philadelphia, PA, USA

8

“Hurrym to the car” Path depends on location of car “Hurry” is a manner of motion verb (Pustejovsky and Moszkowicz, 2011)

slide-9
SLIDE 9

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Events as Programs

  • Path verbs designate a distinguished

value in the state-to-state location change

  • Change in value is tested
  • Manner of motion verbs iterate a state-

to-state location change

  • Change in value is assigned/reassigned
  • Verbs can be realized as programs

enacted over arguments (Naumann, 1999)

August 5, 2016 Philadelphia, PA, USA

9

slide-10
SLIDE 10

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Events as Programs

  • Programs are compositional
  • Program’s linguistic representation can be

broken down into subevents

  • Simple programs
  • translocate, rotate, grasp, hold, release, etc.
  • Complex programs
  • put, stack, etc.

August 5, 2016 Philadelphia, PA, USA

10

slide-11
SLIDE 11

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

VoxML

  • VoxML:

Visual Object Concept Modeling Language (Pustejovsky and Krishnaswamy, 2016)

  • Annotation and modeling language for “voxemes”
  • Visual instantiation of a lexeme
  • Scaffold for mapping from lexical information to

simulated objects and operationalized behaviors

  • Encodes afforded behaviors for each object
  • Gibsonian - afforded by object structure (e.g. grasp,

move, lift) (Gibson, 1977; 1979)

  • Telic - goal-directed, purposeful (e.g. drink from)

(Pustejovsky, 1995)

August 5, 2016 Philadelphia, PA, USA

11

slide-12
SLIDE 12

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

VoxML

August 5, 2016 Philadelphia, PA, USA

12

slide-13
SLIDE 13

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

VoxML

August 5, 2016 Philadelphia, PA, USA

13

slide-14
SLIDE 14

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

VoxSim: Software Architecture

August 5, 2016 Philadelphia, PA, USA

14

We begin by inpu+ng a sentence in plain English

Put the spoon in the mug

slide-15
SLIDE 15

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

VoxSim: Software Architecture

August 5, 2016 Philadelphia, PA, USA

15

From a dependency parse, we extract labeled en<<es in the scene, and verbs those en<<es may afford

Put the spoon in the mug put [in] mug spoon Voxeme: PROGRAM Voxeme: RELATION(OBJECT) Voxeme: OBJECT

slide-16
SLIDE 16

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

VoxSim: Software Architecture

August 5, 2016 Philadelphia, PA, USA

16

Resolve the parsed sentence into a predicate-logic formula

Voxeme: PROGRAM Voxeme: RELATION(OBJECT) Voxeme: OBJECT put spoon [in] mug put(x,y) x := spoon y := in(z) z: = mug put(spoon,in(mug))

slide-17
SLIDE 17

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

VoxSim: Software Architecture

August 5, 2016 Philadelphia, PA, USA

17

Each predicate is opera<onalized according to its type structure

put(spoon,in(mug))

  • in(z): takes object,
  • utputs location
  • put(x,y): path verb
  • while(!at(y), move(x))
slide-18
SLIDE 18

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Semantic Processing

August 5, 2016 Philadelphia, PA, USA

18

  • Object bounds may not

contour to geometry

  • e.g. Concave objects
  • Semantic information

imposes further constraints

  • “in cup”: (PO | TPP | NTPP)

with area denoted by cup’s interior

  • Interpenetrates bounds,

but not geometry

slide-19
SLIDE 19

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Semantic Processing

August 5, 2016 Philadelphia, PA, USA

19

  • Can test be satisfied

with current object configuration?

  • Can test be satisfied

by reorienting

  • bjects?
  • Can test be satisfied

at all?

slide-20
SLIDE 20

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Rig Attachment

August 5, 2016 Philadelphia, PA, USA

20

  • Temporary parent-child relationship between joint on rig

and manipulated object

  • Allows agent and object to move together
  • “Object model” + “Action model” = “Event model”
slide-21
SLIDE 21

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Demo

August 5, 2016 Philadelphia, PA, USA

21

slide-22
SLIDE 22

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Discussion

August 5, 2016 Philadelphia, PA, USA

22

  • Platform for incorporating motion/dynamic semantics

into visualization

  • Visualization → Simulation → Minimal Model
  • Runtime visualization generation necessitates assigning

values in the simulation to parameters unspecified in minimal model

  • e.g. speed, direction, etc.
  • Complete set of primitive programs in a particular

domain unknown

slide-23
SLIDE 23

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Future Work

August 5, 2016 Philadelphia, PA, USA

23

  • Monte-Carlo simulation generation with multiple

evaluation tasks

  • Given visualization with randomly-assigned

underspecified variables, choose best description

  • Given description, choose best visualization from

randomly-generated set

  • Automatic evaluation of actual simulation result vs.

DITL-derived satisfaction conditions

  • Corpus building for linked videos and simulations with

event labels for machine learning of event classification

slide-24
SLIDE 24

Spatial Cognition 2016

Nikhil Krishnaswamy | nkrishna@brandeis.edu

Acknowledgments

August 5, 2016 Philadelphia, PA, USA

24

Brandeis University Student Workers Jessica Huynh Paul Kang Subahu Rayamajhi Amy Wu Beverly Lum Victoria Tran