Weakly Supervised Semantic Parsing with Abstract Examples Omer - PowerPoint PPT Presentation

Weakly Supervised Semantic Parsing with Abstract Examples Omer Goldman , Veronica Latcinnik, Udi Naveh, Amir Globerson, Jonathan Berant Tel Aviv University ACL, July 2018

Semantic Parsing 𝑋 h 𝑏𝑢 𝑗𝑡 𝑢 h 𝑓 𝑑𝑏𝑞𝑗𝑢𝑏𝑚 𝑝𝑔 𝑢 h 𝑓 𝑚𝑏𝑠𝑕𝑓𝑡𝑢 𝑉𝑇 𝑡𝑢𝑏𝑢𝑓 ? model CapitalOf.argmax(Type.State ∩ LocatedIn.US,Population) KB: execution Sacramento � Introduction � Semantic parser � Abstract examples � Results � Conclusions 1

Training with Full Supervision � Training examples: E X P E N S I V E x: y: CapitalOf.argmax(Type.State ∩ LocatedIn.US,Population) $ € ¥ � Introduction � Semantic parser � Abstract examples � Results � Conclusions 2

Training with Weak Supervision � Training examples: x: y: Sacramento � Introduction � Semantic parser � Abstract examples � Results � Conclusions 3

Problems with Weak Supervision � Exponential search space 3+3*15 1 decoding 5+30 2-2 4+60/3 � Introduction � Semantic parser � Abstract examples � Results � Conclusions 4

Problems with Weak Supervision � Spurious programs (Pasupat and Liang, 2016; Guu et al., 2017) 20-16 32/4 decoding 4*2-4 50-1 3+85 Correct program : 2*2 � Introduction � Semantic parser � Abstract examples � Results � Conclusions 4

CNLVR (Suhr et al., 2017) � Introduction � Semantic parser � Abstract examples � Results � Conclusions 6

Semantic Parsing 𝑈 h 𝑓𝑠𝑓 𝑗𝑡 𝑏 𝑐𝑚𝑣𝑓 𝑡𝑟𝑣𝑏𝑠𝑓 model exist(filter(ALL_ITEMS, λ x.IsBlue(x) ∩ IsSquare(x))) KB: execution Binary! True 50% spurious � Introduction � Semantic parser � Abstract examples � Results � Conclusions 7

Insight 𝑈 h 𝑓𝑠𝑓 𝑗𝑡 𝑓𝑦𝑏𝑑𝑢𝑚𝑧 𝑝𝑜𝑓 𝑐𝑚𝑏𝑑𝑙 𝑑𝑗𝑠𝑑𝑚𝑓 𝑜𝑝𝑢 𝑢𝑝𝑣𝑑 h 𝑗𝑜𝑕 𝑢 h 𝑓 𝑓𝑒𝑕𝑓 � Equal( 1 ,(filter(ALL_ITEMS, λ x.IsBlack(x) ∩ IsCircle(x) ∩ ¬IsTouchingWall(x))) � Equal( 1 ,(filter(ALL_ITEMS, λ x.IsBlack(x) ∩ IsCircle(x) ∩ ¬IsTouchingWall(x))) � GreaterEqual( 3 ,(filter(ALL_ITEMS, λ x.IsBlue(x) ∩ IsTriangle(x) ∩ ¬IsTouchingWall(x))) � GreaterEqual( 1 ,(filter(ALL_ITEMS, λ x.IsBlue(x) ∩ IsTriangle(x) ∩ ¬IsTouchingWall(x))) � LessEqual( 3 ,(filter(ALL_ITEMS, λ x.IsYellow(x) ∩ IsRectangle(x) ∩ ¬IsTouchingWall(x))) � Introduction � Semantic parser � Abstract examples � Results � Conclusions 8

Contributions 𝑈 h 𝑓𝑠𝑓 𝑗𝑡 𝑏 𝑧𝑓𝑚𝑚𝑝𝑥 𝑑𝑗𝑠𝑑𝑚𝑓 exist(filter(ALL_ITEMS, λ x.IsYellow(x) ⋀ IsCircle(x))) 𝑈 h 𝑓𝑠𝑓 𝑗𝑡 𝑏 𝐷 − 𝐷𝑃𝑀𝑃𝑆 𝐷 − 𝑇𝐼𝐵𝑄𝐹 exist(filter(ALL_ITEMS, λ x.IsC-COLOR(x) ⋀ IsC-SHAPE(x))) Data augmentation Abstract cache helps search tackles spuriousness CNLVR improvement: 67.8 � 82.5 � Introduction � Semantic parser � Abstract examples � Results � Conclusions 9

Semantic Parsing 11

Logical Program x: z: exist(filter(ALL_BOXES, λ x.ge(3,count(filter(x, λ y.IsBlue(y)))))) Variable Variable Function Set(Item) Item Set() � Bool Constant Function Set(Set(Item)) Function Set() � Int Set() BoolFunc() � Set() � Introduction � Semantic parser � Abstract examples � Results � Conclusions 12

Model Exist(Filter(ALL_ITEMS, λ x.And(IsYellow(x), IsTriangle(x)))) “There is a yellow triangle” Beam Exist(Filter(ALL_ITEMS, λ x.IsYellow(x))) Encoder – search GraeterEqual(1,count(Filter(ALL_ITEMS, λ x.And(IsYellow(x),IsTriangle(x))))) Decoder � Training maximizes log-likelihood of correct programs � + discriminative re-ranker � Introduction � Semantic parser � Abstract examples � Results � Conclusions 13

Abstract Examples 14

Abstraction 𝑈 h 𝑓𝑠𝑓 𝑗𝑡 𝑏 𝑧𝑓𝑚𝑚𝑝𝑥 𝑑𝑗𝑠𝑑𝑚𝑓 𝑈 h 𝑓𝑠𝑓 𝑗𝑡 𝑏 𝑐𝑚𝑣𝑓 𝑡𝑟𝑣𝑏𝑠𝑓 exist(filter(ALL_ITEMS, λ x.IsYellow(x) ⋀ IsCircle(x))) exist(filter(ALL_ITEMS, λ x.IsBlue(x) ⋀ IsRectangle(x))) 𝑈 h 𝑓𝑠𝑓 𝑗𝑡 𝑏 𝐷 − 𝐷𝑃𝑀𝑃𝑆 𝐷 − 𝑇𝐼𝐵𝑄𝐹 exist(filter(ALL_ITEMS, λ x.IsC-COLOR(x) ⋀ IsC-SHAPE(x))) � Introduction � Semantic parser � Abstract examples � Results � Conclusions 15

Analysis 3163 CNLVR sentences There is………… • One of the……. • There are…….. • ~1300 abstract sentences Exactly two….. • There is………… • In two of………. • There is…………… • There is……….. • There are………… • There are…….. • C-Num of…….…… • There is……….. • There is........... • abstraction One square….. • One tower……….. • There is……….. • There are…………. • One of the……. • C-Num C-Shape… • There are…….. • There is……………. • There is…….... • C-Num towers….. • Two towers….. • Another last……… • There are…….. • There is……….. • One circle……. • There is……….. • Last one……….. • � ~150 abstract sentences cover 50% of CNLVR. � Introduction � Semantic parser � Abstract examples � Results � Conclusions 16

Abstraction � Data augmentation � Abstract cache � Introduction � Semantic parser � Abstract examples � Results � Conclusions 17

Data Augmentation 𝑈 h 𝑓𝑠𝑓 𝑗𝑡 𝑏 𝑧𝑓𝑚𝑚𝑝𝑥 𝑑𝑗𝑠𝑑𝑚𝑓 𝑈 h 𝑓𝑠𝑓 𝑗𝑡 𝑏 𝑐𝑚𝑣𝑓 𝑠𝑓𝑑𝑢𝑏𝑜𝑕𝑚𝑓 exist(filter(ALL_ITEMS, λ x.IsBlue(x) ⋀ IsSquare(x))) 𝑈 h 𝑓𝑠𝑓 𝑗𝑡 𝑏 𝑧𝑓𝑚𝑚𝑝𝑥 𝑢𝑠𝑗𝑏𝑜𝑕𝑚𝑓 exist(filter(ALL_ITEMS, λ x.IsYellow(x) ⋀ IsTriangle(x))) 𝑈 h 𝑓𝑠𝑓 𝑗𝑡 𝑏 𝑐𝑚𝑏𝑑𝑙 𝑑𝑗𝑠𝑑𝑚𝑓 exist(filter(ALL_ITEMS, λ x.IsBlack(x) ⋀ IsCircle(x))) 𝑈 h 𝑓𝑠𝑓 𝑗𝑡 𝑏 𝐷 − 𝐷𝑃𝑀𝑃𝑆 𝐷 − 𝑇𝐼𝐵𝑄𝐹 exist(filter(ALL_ITEMS, λ x.IsC-COLOR(x) ⋀ IsC-SHAPE(x))) � Introduction � Semantic parser � Abstract examples � Results � Conclusions 18

Training Procedure ~100 Abstract examples (abs. sent., abs. prog.) Generation ~6000 Instantiated examples Supervised model Supervised (sentence, program) training initialization 3163 CNLVR training examples Weakly-supervised model Weakly (sentence, answer) supervised training � Introduction � Semantic parser � Abstract examples � Results � Conclusions 19

Abstract Cache � Introduction � Semantic parser � Abstract examples � Results � Conclusions 20

Reward Tying 50% spurious � Introduction � Semantic parser � Abstract examples � Results � Conclusions 21

Reward Tying : False : False 6.25% spurious � Introduction � Semantic parser � Abstract examples � Results � Conclusions 22

Results 23

Models Baselines (Suhr et al., 2017) � Majority label ( True ) � Max Entropy classifier on extracted features � Supervised trained model (+Re-ranker) � Weakly supervised trained model (+Re-ranker) � Introduction � Semantic parser � Abstract examples � Results � Conclusions 24

Results - Public test set Test-P Accuracy Test-P Consistency 84 81.7 76.6 67.7 66.9 65 60.1 56.2 51.8 38.3 Majority MaxEnt Sup. Sup.+Rerank W.Sup. W.Sup.+Rerank � Introduction � Semantic parser � Abstract examples � Results � Conclusions 25

Ablations No data ugmentation Abstract weakly pervised parser � Introduction � Semantic parser � Abstract examples � Results � Conclusions 26

Data augmentation addition Ablations 85.7 77.2 71.4 67.4 58.2 56.1 Dev Accuracy Dev Consistency 41.2 7.1 -Abstraction -Data augment. -Beam cache W.Sup.+Rerank Cache addition � Introduction � Semantic parser � Abstract examples � Results � Conclusions 27

Conclusions 28

Conclusions Closed domains Better Abstraction training Weak supervision Facilitate information sharing � Similar ideas in: Dong and Lapata (2018) and Zhang et al. (2017) � Automation would be useful � Introduction � Semantic parser � Abstract examples � Results � Conclusions 29

Thank you https://github.com/udiNaveh/nlvr_tau_nlp_final_proj