Verb Physics Relative Physical Knowledge of Actions and Objects - - PowerPoint PPT Presentation

▶
verb physics
SMART_READER_LITE
LIVE PREVIEW

Verb Physics Relative Physical Knowledge of Actions and Objects - - PowerPoint PPT Presentation

Jun 04, 2023 422 likes •1.17k views

QUALS EDITION Verb Physics Relative Physical Knowledge of Actions and Objects Max Forbes Yejin Choi [Gao et al., 2016] [Angeli and Manning, 2014] [Gordon and Schubert, 2012] [Li et al., 2014] Physical properties of objects What is the

slide-1
SLIDE 1

Verb Physics

Relative Physical Knowledge of Actions and Objects

Max Forbes Yejin Choi

QUALS EDITION

slide-2
SLIDE 2

[Angeli and Manning, 2014] [Li et al., 2014] [Gordon and Schubert, 2012] [Gao et al., 2016]

slide-3
SLIDE 3

What is the physical world like? How big are dogs? Tennis balls? Cars? If I drop this styrofoam ball into the steel table, will either break? Physical properties of objects size strength

slide-4
SLIDE 4

“I am larger than a chair”

slide-5
SLIDE 5

“I am larger than a chair”

slide-6
SLIDE 6

“I am larger than a chair” “I am larger than a ball” “I am larger than a stone” “I am larger than a pen” “I am larger than a towel”

[Misra et al., 2016] [Sorower et al., 2011] [Grice, 1975]

slide-7
SLIDE 7

“The horse was as small as a dog!”

⟹ horse =size dog ?

slide-8
SLIDE 8

“Hey robot, pass me the <unk>.” “OK.” (attempts to pick up table)

slide-9
SLIDE 9

“I picked up the <thing>.” “I took a drink from the <thing>.” “The <thing> shattered when it hit the ground

slide-10
SLIDE 10

Two related problems

Physical properties implied by predicates

“I picked up the <thing>.” “I took a drink from the <thing>.” “The <thing> shattered when it hit the ground

Physical properties of objects

size weight strength

slide-11
SLIDE 11
  • 1. Introduction
  • 2. Related work
  • 3. Approach
  • 4. Model
  • 5. Data
  • 6. Evaluation
slide-12
SLIDE 12

Pattern-based IE “how often do you sleep?”

[Gordon et al., 2010] [Gordon and Schubert, 2012]

Word embeddings “is yellow” “is large”

[Rubinstein et al., 2015]

Commonsense knowledge base completion

[Li et al., 2016] [Angeli and Manning, 2013] [Angeli and Manning, 2014]

“not all birds can fly”

slide-13
SLIDE 13

Verbs grounded in robotics + vision

[Gao et al., 2016] [She and Chai, 2016] [Misra et al., 2014]

Semantic proto-roles

[Dowty, 1991] [Kako, 2006] [Reisinger et al., 2015]

Overcoming reporting bias

[Misra et al., 2016] [Sorower et al., 2011] [Tellex et al., 2011]

“cutting changes the number of pieces”

slide-14
SLIDE 14
  • 1. Introduction
  • 2. Related work
  • 3. Approach
  • 4. Model
  • 5. Data
  • 6. Evaluation
slide-15
SLIDE 15

Two related problems

Physical properties implied by predicates

“I picked up the <unk>.” “I took a drink from the <unk>.” “The <unk> shattered when it hit the ground

Physical properties of objects

size weight strength

slide-16
SLIDE 16

Attributes

x >size y x >weight y x <rigidness y x >strength y x >speed y

slide-17
SLIDE 17

“I threw the _____”

slide-18
SLIDE 18

“I threw the _____”

ball stone chair

slide-19
SLIDE 19

“I threw the _____”

ball stone chair game party

slide-20
SLIDE 20

“I threw the _____”

ball stone chair

slide-21
SLIDE 21

x threw y

slide-22
SLIDE 22

x threw y

x is bigger than y

slide-23
SLIDE 23

x threw y

x is bigger than y x weighs more than y as a result, y will be moving faster than x

slide-24
SLIDE 24

⟹ x >size y

x threw y

⟹ x >weight y ⟹ x <speed y

Action frame

slide-25
SLIDE 25

Terminology

Action frames — simple syntax-based verb constructions that compare two objects

slide-26
SLIDE 26

Terminology

Action frames

x threw y

PERSON threw x into y PERSON threw on x — simple syntax-based verb constructions distinct action frames for the same verb that compare two objects

slide-27
SLIDE 27

Terminology

Action frames

x threw y

PERSON threw x into y PERSON threw on x Objects — simple syntax-based verb constructions — non-abstract nouns that compare two objects

ball train evil time

✓ ✓

✘ ✘

slide-28
SLIDE 28

Two related problems

Physical properties implied by predicates

“I picked up the <thing>.” “I took a drink from the <thing>.” “The <thing> shattered when it hit the ground

Physical properties of objects

size weight strength

slide-29
SLIDE 29

Two related problems

Physical properties implied by predicates

“I picked up the <unk>.” “I took a drink from the <unk>.” “The <unk> shattered when it hit the ground

Physical properties of objects

size weight strength F = “x threw y” attribute: size correct value: > Example intuition: “x threw y” ⟹ x >size y

takes values in {>, <, '}

slide-30
SLIDE 30

Two related problems

Physical properties implied by predicates

“I picked up the <unk>.” “I took a drink from the <unk>.” “The <unk> shattered when it hit the ground

Physical properties of objects

size weight strength F = “x threw y” attribute: size correct value: > Example intuition: “x threw y” ⟹ x >size y

takes values in {>, <, '}

Example attribute: size correct value: > = (person, ball) intuition: people are generally larger than balls

Jp,q

takes values in {>, <, '}

slide-31
SLIDE 31

FRAME KNOWLEDGE

x threw y

Solving both puzzles together

Action frame

slide-32
SLIDE 32

FRAME KNOWLEDGE OBJECT KNOWLEDGE

person, ball person, stone person, chair

x threw y

Solving both puzzles together

slide-33
SLIDE 33

FRAME KNOWLEDGE OBJECT KNOWLEDGE

person, ball person, stone person, chair

person >size ball person >size stone person >size chair

⟹ x >size y

x threw y

Solving both puzzles together

slide-34
SLIDE 34

FRAME KNOWLEDGE OBJECT KNOWLEDGE

person, ball person, stone person, chair

person >size ball person >size stone person >size chair

⟹ x >size y

x threw y

Solving both puzzles together

slide-35
SLIDE 35

FRAME KNOWLEDGE OBJECT KNOWLEDGE

person, ball person, stone person, chair

person >size ball person >size stone person >size chair

⟹ x >size y

x threw y OBSERVABLE IN LANGUAGE (!)

slide-36
SLIDE 36
  • 1. Introduction
  • 2. Related work
  • 3. Approach
  • 4. Model
  • 5. Data
  • 6. Evaluation
slide-37
SLIDE 37

ACTION FRAMES OBJECT PAIRS

High level model

slide-38
SLIDE 38

ACTION FRAMES OBJECT PAIRS

High level model

slide-39
SLIDE 39

ACTION FRAMES OBJECT PAIRS

High level model

slide-40
SLIDE 40

ACTION FRAMES OBJECT PAIRS

High level model

slide-41
SLIDE 41

ACTION FRAMES OBJECT PAIRS

F a

vt

Random variables Take values in {>, <, '}

slide-42
SLIDE 42

ACTION FRAMES OBJECT PAIRS

F a

vt

Random variables Take values in {>, <, '}

F size

threw1 ≈ “x threw y”

slide-43
SLIDE 43

ACTION FRAMES OBJECT PAIRS

F a

vt

Random variables Take values in

p(F size

threw1 = >) := p(“x threw y” ⇒ x >size y)

{>, <, '} F size

threw1 ≈ “x threw y”

slide-44
SLIDE 44

ACTION FRAMES OBJECT PAIRS

F a

vt

Random variables Random variables Take values in Take values in

{>, <, '} {>, <, '} Ja

p,q

slide-45
SLIDE 45

ACTION FRAMES OBJECT PAIRS

F a

vt

Random variables Random variables Take values in Take values in

{>, <, '} {>, <, '} (person, ball) Ja

p,q

Jsize

person,ball ≈

slide-46
SLIDE 46

ACTION FRAMES OBJECT PAIRS

F a

vt

Random variables Random variables Take values in Take values in

{>, <, '} {>, <, '} (person, ball) Jsize

person,ball ≈

Ja

p,q

p(Jsize

person,ball = >) := p(person >size ball)

slide-47
SLIDE 47

ACTION FRAMES OBJECT PAIRS

F a

vt

Random variables Random variables Take values in Take values in

{>, <, '} {>, <, '} Ja

p,q

slide-48
SLIDE 48

ACTION FRAMES OBJECT PAIRS

F a

vt

Random variables Random variables Take values in Take values in

{>, <, '} {>, <, '} Ja

p,q

slide-49
SLIDE 49

size

stone rock

Jperson, Jperson,

house

Jperson, Object pair random variables

slide-50
SLIDE 50

size

stone rock

Jperson, Jperson,

house

Jperson,

ψo

Object similarity binary factors

slide-51
SLIDE 51

ψo

size vwalk

stone rock

Jperson, Jperson,

house

Jperson, vsquish vthrow vwalk

ψv

Action frames grouped by verb Verb similarity binary factors

slide-52
SLIDE 52

ψv ψo

size vthrow vsquish vwalk

stone rock

Jperson, Jperson,

house

Jperson,

Fthrow1 Fthrow2 Fthrow3

ψf

Several action frames per verb Similar frame construction binary factor

slide-53
SLIDE 53

ψv ψo ψs

size vthrow vsquish vwalk

stone rock Fthrow1 Fthrow2 Fthrow3 ψf

Jperson, Jperson,

house

Jperson, Action-object compatibility binary factors

slide-54
SLIDE 54

… … …

ψv ψo

size

weight

strength

vthrow vsquish vwalk

stone rock Fthrow1 Fthrow2 Fthrow3 ψf

… Jperson, Jperson,

house

Jperson,

ψs

vsquish Jperson,

stone house

Jperson, vthrow More attributes

size size size size size size size size size weight weight strength strength

ψa ψa

Similar attribute binary factors

slide-55
SLIDE 55

v

f

µ

ψf

µ

Loopy belief propagation

ACTION FRAMES OBJECT PAIRS

slide-56
SLIDE 56
  • 1. Introduction
  • 2. Related work
  • 3. Approach
  • 4. Model
  • 5. Data
  • 6. Evaluation
slide-57
SLIDE 57

ACTION FRAMES OBJECT PAIRS

Why collect data?

slide-58
SLIDE 58

ACTION FRAMES OBJECT PAIRS

Why collect data?

slide-59
SLIDE 59
  • Small seed set (5%) breaks symmetry
  • Evaluate generalizability (dev = 45%, test = 50%)

ACTION FRAMES OBJECT PAIRS

Why collect data?

slide-60
SLIDE 60

Verbs

  • took
  • grew
  • washed
  • trimmed
  • squished
  • got
  • looked
  • wrote
  • entered
  • kept
  • lived
  • played
  • …

Selecting frames and objects

“Action” verbs

[Levin, 1993]

slide-61
SLIDE 61

Verbs

  • took
  • grew
  • washed
  • trimmed
  • squished
  • got
  • looked
  • wrote
  • entered
  • kept
  • lived
  • played
  • …

Action frames

Selecting frames and objects

Syntax + surface + crowdsourcing

  • x squished y
  • x squished on y
  • PERSON squished

x with y

  • PERSON squished

x on y … …

slide-62
SLIDE 62

Action frames Object pairs

Selecting frames and objects

  • x squished y
  • x squished on y
  • PERSON squished

x with y

  • PERSON squished

x on y … …

  • spider, boot
  • spider, glee

… …

PMI > 0 on Google Syntax Ngrams not abstract via Wordnet [Goldberg and Orwant, 1993] [Miller, 1995]

Verbs

  • took
  • grew
  • washed
  • trimmed
  • squished
  • got
  • looked
  • wrote
  • entered
  • kept
  • lived
  • played
  • …
slide-63
SLIDE 63

Total Seed / dev / test Verbs 100 5 / 45 / 50 Frames 813 65 / 333 / 415 Object pairs 3656 183 / 1645 / 1828

Data statistics

~200 distinct

  • bjects

~8 action frames / verb

slide-64
SLIDE 64
  • 1. Introduction
  • 2. Related work
  • 3. Approach
  • 4. Model
  • 5. Data
  • 6. Evaluation
slide-65
SLIDE 65

ACCURACY (TEST) 0.2 0.4 0.6 0.8 R A N D O M N G R A M S M A J O R I T Y R A N D O M N G R A M S M A J O R I T Y

0.51 0.44 0.33 0.33 0.33 0.33

ACTION FRAMES OBJECTS

slide-66
SLIDE 66

ACCURACY (TEST) 0.2 0.4 0.6 0.8 R A N D O M N G R A M S M A J O R I T Y E M B

  • M

A X E N T R A N D O M N G R A M S M A J O R I T Y E M B

  • M

A X E N T

0.66 0.66 0.51 0.44 0.33 0.33 0.33 0.33

ACTION FRAMES OBJECTS

slide-67
SLIDE 67

ACCURACY (TEST) 0.2 0.4 0.6 0.8 R A N D O M N G R A M S M A J O R I T Y E M B

  • M

A X E N T O U R M O D E L R A N D O M N G R A M S M A J O R I T Y E M B

  • M

A X E N T O U R M O D E L

0.70 0.75 0.66 0.66 0.51 0.44 0.33 0.33 0.33 0.33

ACTION FRAMES OBJECTS

slide-68
SLIDE 68

Correct dev set examples

___ opened ___ “She opened the jar of peanut butter.”

she >size jar

✓ size

slide-69
SLIDE 69

PERSON set ___ upon ___ “He set the kettle upon the stove.”

Correct dev set examples

kettle <weight stove

✓ weight

slide-70
SLIDE 70

Incorrect dev set examples

polysemy

“She caught the runner in first.” ___ caught ___ “She caught the baseball.”

she >speed runner

  • ur model

she <speed baseball

ground truth

speed

slide-71
SLIDE 71

complex physics

“He stopped a fly with a jar.” “She stopped the car with the brake.” PERSON stopped ___with ___

Incorrect dev set examples

fly <weight jar

  • ur model

car >weight brake

ground truth

weight

slide-72
SLIDE 72

Summary

  • Reverse engineer

commonsense physical knowledge

  • Overcome reporting

bias by modeling frames and objects

Max Forbes Yejin Choi {mbforbes,yejin}@cs.uw.edu

slide-73
SLIDE 73

Summary

  • Reverse engineer

commonsense physical knowledge

  • Overcome reporting

bias by modeling frames and objects

Max Forbes Yejin Choi {mbforbes,yejin}@cs.uw.edu

VerbPhysics

uwnlp.github.io/verbphysics/

  • New dataset