Weakly Supervised Semantic Parsing with Abstract Examples Omer - - PowerPoint PPT Presentation

β–Ά
weakly supervised semantic parsing with abstract examples
SMART_READER_LITE
LIVE PREVIEW

Weakly Supervised Semantic Parsing with Abstract Examples Omer - - PowerPoint PPT Presentation

Jan 19, 2024 148 likes β€’464 views

Weakly Supervised Semantic Parsing with Abstract Examples Omer Goldman , Veronica Latcinnik, Udi Naveh, Amir Globerson, Jonathan Berant Tel Aviv University ACL, July 2018 Semantic Parsing h h

slide-1
SLIDE 1

Weakly Supervised Semantic Parsing with Abstract Examples

Omer Goldman, Veronica Latcinnik, Udi Naveh, Amir Globerson, Jonathan Berant Tel Aviv University ACL, July 2018

slide-2
SLIDE 2

Semantic Parsing

model

CapitalOf.argmax(Type.State ∩ LocatedIn.US,Population)

𝑋 h𝑏𝑒 𝑗𝑑 𝑒h𝑓 π‘‘π‘π‘žπ‘—π‘’π‘π‘š 𝑝𝑔 𝑒h𝑓 π‘šπ‘π‘ π‘•π‘“π‘‘π‘’ 𝑉𝑇 𝑑𝑒𝑏𝑒𝑓? execution

Sacramento

Introduction Semantic parser Abstract examples Results Conclusions

KB:

1

slide-3
SLIDE 3

Training with Full Supervision

Training examples:

Introduction Semantic parser Abstract examples Results Conclusions

x: y: CapitalOf.argmax(Type.State ∩ LocatedIn.US,Population)

E X P E N S I V E

$ € Β₯

2

slide-4
SLIDE 4

Training with Weak Supervision

Training examples:

Introduction Semantic parser Abstract examples Results Conclusions

x: y: Sacramento

3

slide-5
SLIDE 5

Problems with Weak Supervision

Exponential search space

Introduction Semantic parser Abstract examples Results Conclusions

1 3+3*15 5+30 2-2 4+60/3

decoding

4

slide-6
SLIDE 6

Problems with Weak Supervision

Spurious programs (Pasupat and Liang, 2016; Guu et al., 2017)

Introduction Semantic parser Abstract examples Results Conclusions

32/4 20-16 4*2-4 50-1 3+85

decoding

4

Correct program: 2*2

slide-7
SLIDE 7

CNLVR (Suhr et al., 2017)

Introduction Semantic parser Abstract examples Results Conclusions 6

slide-8
SLIDE 8

Semantic Parsing

model

exist(filter(ALL_ITEMS,λx.IsBlue(x) ∩ IsSquare(x)))

π‘ˆh𝑓𝑠𝑓 𝑗𝑑 𝑏 π‘π‘šπ‘£π‘“ π‘‘π‘Ÿπ‘£π‘π‘ π‘“ execution True

Introduction Semantic parser Abstract examples Results Conclusions

KB:

Binary!

50% spurious

7

slide-9
SLIDE 9

Insight

Introduction Semantic parser Abstract examples Results Conclusions

π‘ˆh𝑓𝑠𝑓 𝑗𝑑 π‘“π‘¦π‘π‘‘π‘’π‘šπ‘§ π‘π‘œπ‘“ π‘π‘šπ‘π‘‘π‘™ π‘‘π‘—π‘ π‘‘π‘šπ‘“ π‘œπ‘π‘’ 𝑒𝑝𝑣𝑑hπ‘—π‘œπ‘• 𝑒h𝑓 𝑓𝑒𝑕𝑓

Equal(1,(filter(ALL_ITEMS,λx.IsBlack(x) ∩ IsCircle(x) ∩ ¬IsTouchingWall(x))) GreaterEqual(3,(filter(ALL_ITEMS,λx.IsBlue(x) ∩ IsTriangle(x) ∩ ¬IsTouchingWall(x))) GreaterEqual(1,(filter(ALL_ITEMS,λx.IsBlue(x) ∩ IsTriangle(x) ∩ ¬IsTouchingWall(x))) LessEqual(3,(filter(ALL_ITEMS,λx.IsYellow(x) ∩ IsRectangle(x) ∩ ¬IsTouchingWall(x))) Equal(1,(filter(ALL_ITEMS,λx.IsBlack(x) ∩ IsCircle(x) ∩ ¬IsTouchingWall(x)))

8

slide-10
SLIDE 10

Contributions

π‘ˆh𝑓𝑠𝑓 𝑗𝑑 𝑏 𝐷 βˆ’ 𝐷𝑃𝑀𝑃𝑆 𝐷 βˆ’ 𝑇𝐼𝐡𝑄𝐹

exist(filter(ALL_ITEMS,Ξ»x.IsC-COLOR(x)β‹€IsC-SHAPE(x)))

π‘ˆh𝑓𝑠𝑓 𝑗𝑑 𝑏 π‘§π‘“π‘šπ‘šπ‘π‘₯ π‘‘π‘—π‘ π‘‘π‘šπ‘“

exist(filter(ALL_ITEMS,Ξ»x.IsYellow(x)β‹€IsCircle(x)))

Data augmentation

helps search

Abstract cache

tackles spuriousness

CNLVR improvement: 67.8 82.5

Introduction Semantic parser Abstract examples Results Conclusions 9

slide-11
SLIDE 11

Semantic Parsing

11

slide-12
SLIDE 12

Logical Program

Introduction Semantic parser Abstract examples Results Conclusions

x: z: exist(filter(ALL_BOXES,Ξ»x.ge(3,count(filter(x,Ξ»y.IsBlue(y))))))

Constant Set(Set(Item)) Function Set() Int Function Set() Bool Variable Set(Item) Variable Item Function Set() BoolFunc() Set() 12

slide-13
SLIDE 13

Model

Training maximizes log-likelihood of correct programs + discriminative re-ranker

Introduction Semantic parser Abstract examples Results Conclusions

β€œThere is a yellow triangle”

Exist(Filter(ALL_ITEMS,Ξ»x.IsYellow(x))) Exist(Filter(ALL_ITEMS,Ξ»x.And(IsYellow(x), IsTriangle(x)))) GraeterEqual(1,count(Filter(ALL_ITEMS,Ξ»x.And(IsYellow(x),IsTriangle(x)))))

Encoder – Decoder Beam search

13

slide-14
SLIDE 14

Abstract Examples

14

slide-15
SLIDE 15

Abstraction

π‘ˆh𝑓𝑠𝑓 𝑗𝑑 𝑏 π‘π‘šπ‘£π‘“ π‘‘π‘Ÿπ‘£π‘π‘ π‘“ π‘ˆh𝑓𝑠𝑓 𝑗𝑑 𝑏 π‘§π‘“π‘šπ‘šπ‘π‘₯ π‘‘π‘—π‘ π‘‘π‘šπ‘“

π‘ˆh𝑓𝑠𝑓 𝑗𝑑 𝑏 𝐷 βˆ’ 𝐷𝑃𝑀𝑃𝑆 𝐷 βˆ’ 𝑇𝐼𝐡𝑄𝐹 exist(filter(ALL_ITEMS,Ξ»x.IsC-COLOR(x)β‹€IsC-SHAPE(x))) Introduction Semantic parser Abstract examples Results Conclusions

exist(filter(ALL_ITEMS,Ξ»x.IsYellow(x)β‹€IsCircle(x))) exist(filter(ALL_ITEMS,Ξ»x.IsBlue(x)β‹€IsRectangle(x)))

15

slide-16
SLIDE 16

Analysis

~150 abstract sentences cover 50% of CNLVR.

Introduction Semantic parser Abstract examples Results Conclusions

  • There is…………
  • One of the…….
  • There are……..
  • Exactly two…..
  • There is…………
  • In two of……….
  • There is………..
  • There are……..
  • There is………..
  • One square…..
  • There is………..
  • One of the…….
  • There are……..
  • There is……....
  • Two towers…..
  • There are……..
  • There is………..
  • One circle…….
  • There is………..
  • Last one………..

abstraction

  • There is……………
  • There are…………
  • C-Num of…….……
  • There is...........
  • One tower………..
  • There are………….
  • C-Num C-Shape…
  • There is…………….
  • C-Num towers…..
  • Another last………

3163 CNLVR sentences ~1300 abstract sentences

16

slide-17
SLIDE 17

Abstraction

Data augmentation Abstract cache

Introduction Semantic parser Abstract examples Results Conclusions 17

slide-18
SLIDE 18

Data Augmentation

π‘ˆh𝑓𝑠𝑓 𝑗𝑑 𝑏 π‘§π‘“π‘šπ‘šπ‘π‘₯ π‘‘π‘—π‘ π‘‘π‘šπ‘“

π‘ˆh𝑓𝑠𝑓 𝑗𝑑 𝑏 𝐷 βˆ’ 𝐷𝑃𝑀𝑃𝑆 𝐷 βˆ’ 𝑇𝐼𝐡𝑄𝐹

exist(filter(ALL_ITEMS,Ξ»x.IsC-COLOR(x)β‹€IsC-SHAPE(x)))

π‘ˆh𝑓𝑠𝑓 𝑗𝑑 𝑏 π‘π‘šπ‘£π‘“ π‘ π‘“π‘‘π‘’π‘π‘œπ‘•π‘šπ‘“

exist(filter(ALL_ITEMS,Ξ»x.IsBlue(x)β‹€IsSquare(x)))

π‘ˆh𝑓𝑠𝑓 𝑗𝑑 𝑏 π‘§π‘“π‘šπ‘šπ‘π‘₯ π‘’π‘ π‘—π‘π‘œπ‘•π‘šπ‘“

exist(filter(ALL_ITEMS,Ξ»x.IsYellow(x)β‹€IsTriangle(x)))

π‘ˆh𝑓𝑠𝑓 𝑗𝑑 𝑏 π‘π‘šπ‘π‘‘π‘™ π‘‘π‘—π‘ π‘‘π‘šπ‘“

exist(filter(ALL_ITEMS,Ξ»x.IsBlack(x)β‹€IsCircle(x)))

Introduction Semantic parser Abstract examples Results Conclusions 18

slide-19
SLIDE 19

Training Procedure

~100 Abstract examples ~6000 Instantiated examples Supervised model 3163 CNLVR training examples Weakly-supervised model

Introduction Semantic parser Abstract examples Results Conclusions

Generation Supervised training Weakly supervised training initialization

(abs. sent., abs. prog.) (sentence, program) (sentence, answer)

19

slide-20
SLIDE 20

Abstract Cache

Introduction Semantic parser Abstract examples Results Conclusions 20

slide-21
SLIDE 21

Reward Tying

Introduction Semantic parser Abstract examples Results Conclusions

50% spurious

21

slide-22
SLIDE 22

:False :False

6.25% spurious Introduction Semantic parser Abstract examples Results Conclusions

Reward Tying

22

slide-23
SLIDE 23

Results

23

slide-24
SLIDE 24

Models

Majority label (True) Max Entropy classifier on extracted features Supervised trained model (+Re-ranker) Weakly supervised trained model (+Re-ranker)

Introduction Semantic parser Abstract examples Results Conclusions

Baselines (Suhr et al., 2017)

24

slide-25
SLIDE 25

Results - Public test set

Introduction Semantic parser Abstract examples Results Conclusions

Majority MaxEnt Sup. Sup.+Rerank W.Sup. W.Sup.+Rerank

65 60.1 51.8 38.3 84 81.7 76.6 66.9 67.7 56.2

Test-P Accuracy Test-P Consistency

25

slide-26
SLIDE 26

Ablations

Introduction Semantic parser Abstract examples Results Conclusions

No data ugmentation Abstract weakly pervised parser

26

slide-27
SLIDE 27

Ablations

Introduction Semantic parser Abstract examples Results Conclusions

  • Abstraction
  • Data augment.
  • Beam cache

W.Sup.+Rerank

67.4 56.1 41.2 7.1 85.7 77.2 71.4 58.2

Dev Accuracy Dev Consistency

Cache addition Data augmentation addition

27

slide-28
SLIDE 28

Conclusions

28

slide-29
SLIDE 29

Conclusions

Similar ideas in: Dong and Lapata (2018) and Zhang et al. (2017) Automation would be useful

Introduction Semantic parser Abstract examples Results Conclusions 29

Closed domains Weak supervision Abstraction

Better training Facilitate information sharing

slide-30
SLIDE 30

Thank you

https://github.com/udiNaveh/nlvr_tau_nlp_final_proj