Katholieke Universiteit Leuven Computer Science Department LREC May - - PowerPoint PPT Presentation

▶
katholieke universiteit leuven computer science
SMART_READER_LITE
LIVE PREVIEW

Katholieke Universiteit Leuven Computer Science Department LREC May - - PowerPoint PPT Presentation

Apr 28, 2023 301 likes •514 views

Spatial Role Labeling: Task Definition and Annotation Scheme Parisa Kordjamshidi Martijn Van Otterlo Marie-Francine Moens Katholieke Universiteit Leuven Computer Science Department LREC May 2010 1 Introduction Problem Setting

slide-1
SLIDE 1

Katholieke Universiteit Leuven Computer Science Department

Spatial Role Labeling: Task Definition and Annotation Scheme

Parisa Kordjamshidi Martijn Van Otterlo Marie-Francine Moens

LREC May 2010 1

slide-2
SLIDE 2

Introduction

  • Problem Setting
  • Multimodal environment
  • Unrestricted language
  • Machine learning
  • Task definition (Spatial role labeling)
  • Lack of and difficulty of making Data
  • Annotation scheme

2

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-3
SLIDE 3

Motivation

  • 1. so from here exactly opposite is

my desk.

  • 2. and next to that left of that is my

computer, perhaps a meter away.

  • 3. (breathing) ähm.
  • 4. next to that at the wall is my

kitchen, first there is my fridge all the way to the right. Room Description [Bateman, et.al, 2006]

3

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-4
SLIDE 4

Related schemes

  • SpatialML
  • Generalized upper model (GUM)
  • Geographical scheme (Q. Shen, et.al. 2009)
  • Spatial temporal markup(STM) (Pustejovsky and Moszkowicz, 2009)
  • ...

4

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-5
SLIDE 5

Spatial role labeling task

ON(book,table) IN(table, living room) IN(book,living room)?

Give me that book on the table in the living room. Spatial Indicator Trajectory Landmark Spatial Indicator Landmark Trajectory

5

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-6
SLIDE 6

Labeling the parse tree

6

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-7
SLIDE 7

Holistic spatial semantics

Trajector Landmark Direction Region Path Motion Frame of Reference

The entity whose location or motion is of relevance. The reference entity in relation to which the location or motion of the trajector is determined. In terms of its beginning, middle and end. A region of space which is defined in relation to a landmark. The direction along the axes provided by the different frames of reference. A binary component; whether there is perceived motion or not. Three types of frame of reference: intrinsic, relative or absolute.

7

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-8
SLIDE 8

Relational representation

TRAJECTOR(id, token) LANDMARK(id, token, path) SPATIAL-INDICATOR(id, token,general-type, specific-type, spatial-value) MOTION-INDICATOR(id, token) SR(id, trajector, landmark, spatial-indicator, frame-of-reference, motion-indicator)

8

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-9
SLIDE 9
  • a. She is at school.

<TRAJECTOR id=ʼ1ʼ> She </TRAJECTOR>

Trajector

TRAJECTOR(1, she).

9

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-10
SLIDE 10
  • a. The balloon passed over the house.

<LANDMARK id=ʼ1ʼ path=ʼZEROʼ>the house </ LANDMARK>

Landmark

Path

LANDMARK(1, the house, ZERO).

10

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-11
SLIDE 11
  • b. John is in the room.

<SPATIAL-INDICATOR id=ʼ1ʼ general-type=ʼREGIONʼ specific-type=ʼRCC8ʼ spatial-value=ʼTPPʼ > in </SPATIAL-

INDICATOR>

Direction Region

  • a. He is in front of the bush.

<SPATIAL-INDICATOR id=ʼ1ʼ general-type=ʼDIRECTIONʼ specific-type=ʼRELATIVEʼ spatial-value=ʼFRONTʼ> in front

  • f</SPATIAL-INDICATOR>

SPATIAL-INDICATOR(1, in front of,DIRECTION, RELATIVE, FRONT) SPATIAL-INDICATOR(1, in ,REGION, RCC8, TPP)

11

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-12
SLIDE 12

a.The bird flew to its nest.

<MOTION-INDICATOR id=ʼ1ʼ > flew to </MOTIONINDICATOR>

Motion

MOTION-INDICATOR(1, flew to)

12

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-13
SLIDE 13

Spatial relation (SR)

Frame of Reference

+

<TRAJECTOR id=ʼ1ʼ > She</TRAJECTOR>

<LANDMARK id=ʼ1ʼ path=ʼENDʼ> school </LANDMARK> <SPATIAL-INDICATOR id=ʼ1ʼ general-type=ʼREGIONʼ specific-type= ʼRCC8ʼ spatial- value=ʼTPPʼ > to </SPATIALINDICATOR> <MOTION-INDICATOR id=ʼ1ʼ > went to </MOTIONINDICATOR> <SR id=ʼ1ʼ trajector=ʼ1ʼ landmark=ʼ1ʼ spatial-indicator=ʼ1ʼ frame-of-reference=ʼINTRINSICʼ motion-indicator=ʼ1ʼ/>

She went to school.

13

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-14
SLIDE 14

Mapping to formal spatial relations

Region Direction Distance Absolute Relative Qualitative Quantitative RCC8 DC EC TPP-1 EQ PO NTPP TPP NTPP-1 Spatial Relations South SE SW West SW NW North East LEFT RIGHT FRONT BEHIND ABOVE BELOW .... close .... 30 km

14

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-15
SLIDE 15
  • b. John is in the room.

<SPATIAL-INDICATOR id=ʼ1ʼ general-type=ʼREGIONʼ specific-type=ʼRCC8ʼ spatial-value=ʼTPPʼ > in </SPATIAL-INDICATOR>

Direction Region

  • a. He is in front of the bush.

<SPATIAL-INDICATOR id=ʼ1ʼ general-type=ʼDIRECTIONʼ specific-type=ʼRELATIVEʼ spatial-value=ʼFRONTʼ> in front of </SPATIAL-INDICATOR>

SPATIAL-INDICATOR(1, in front of,DIRECTION, RELATIVE, FRONT) SPATIAL-INDICATOR(1, in ,REGION, RCC8, TPP)

Mapping to formal spatial relations

15

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-16
SLIDE 16

Mapping to formal spatial relations

<TRAJECTOR id=ʼ1ʼ > She</TRAJECTOR>

<LANDMARK id=ʼ1ʼ path=ʼENDʼ> school </LANDMARK> <SPATIAL-INDICATOR id=ʼ1ʼ general-type=ʼREGIONʼ specific-type=

ʼRCC8ʼ spatial-value=ʼTPPʼ > to </SPATIALINDICATOR>

<MOTION-INDICATOR id=ʼ1ʼ > went to </MOTIONINDICATOR> <SR id=ʼ1ʼ trajector=ʼ1ʼ landmark=ʼ1ʼ spatial-indicator=ʼ1ʼ frame-of-reference=ʼINTRINSICʼ motion-indicator=ʼ1ʼ/>

She went to school.

16

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-17
SLIDE 17

More complex descriptions

The vase is in the living room, on the table under the window. The man came from between the shops, ran along the road and disappeared down the alley by the church. Behind the shops is a church, to the left of the church is the town hall, in front of the town hall is a fountain.

Sentence level: Discourse level: so from here exactly opposite is my desk. and next to that left of that is my computer, perhaps a meter away. (breathing) ähm, . next to that at the wall is my kitchen, first there is my fridge all the way to the right.

desk desk computer in kitchen

I: Complex locative statements III: Path and route descriptions II: Sequential scene descriptions

17

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-18
SLIDE 18

The vase is still in the living room, on the table under the window. <TRAJECTOR id=ʼ1ʼ > The vase <TRAJECTOR\> <LANDMARK id=ʼ1ʼ path=ʼZEROʼ> the living room <LANDMARK \> <LANDMARK id=ʼ2ʼ path=ʼZEROʼ> the table <LANDMARK \> <LANDMARK id=ʼ3ʼ path=ʼZEROʼ>the window <LANDMARK \> <SPATIAL-INDICATOR id=ʼ1ʼ general-type=ʼREGIONʼ specific-type=ʼRCC8ʼ spatial-value=ʼNTPPʼ > in <SPATIAL-INDICATOR \> <SPATIAL-INDICATOR id=ʼ2ʼ general-type=ʼREGIONʼ specific-type= ʻRCC8ʼ spatial-value=ʼECʼ > on <SPATIAL-INDICATOR \> <SPATIAL-INDICATOR id=ʼ3ʼ general-type=ʼDIRECTIONʼ specific-type= ʻRELATIVEʼ spatial-value=ʼBELOWʼ > under <SPATIAL-INDICATOR \> <SR id=ʼ1ʼ trajector=ʼ1ʼ landmark=ʼ1ʼ spatial-indicator=ʼ1ʼ frame-of-reference=ʼINTRINSICʼ motion-indicator=ʼNILʼ\> <SR id=ʼ2ʼ trajector=ʼ1ʼ landmark=ʼ2ʼ spatial-indicator=ʼ2ʼ frame-of-reference=ʼINTRINSICʼ motion-indicator=ʼNILʼ\> <SR id=3 trajector=1 landmark=3 spatial-indicator=3 frame-of-reference=INTRINSIC motion- indicator=NIL\>

18

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-19
SLIDE 19

Conclusion & Future directions

  • Conclusion
  • General definition of the task
  • Language-independent scheme
  • Covering spatial semantics including dynamic and static

spatial information

  • Ongoing work and Future directions
  • Getting annotated corpus
  • Machine learning (Statistical relational learning)
  • Spatial reasoning, combining multimodal information

19

Spatial Role Labeling: Task Definition and Annotation Scheme

slide-20
SLIDE 20

Questions? Thank you !

20

Spatial Role Labeling: Task Definition and Annotation Scheme