Alignment-Based Composi2onal Seman2cs for Instruc2on Following - PowerPoint PPT Presentation

Alignment-Based Composi2onal Seman2cs for Instruc2on Following Jacob Andreas and Dan Klein UC Berkeley


 Following instruc2ons go down the green hall go all the way then face the chair


 Following instruc2ons go down the green hall go all the way then face the chair


 Following instruc2ons: parsing go down the green hall go all the way then face the chair


 Following instruc2ons: parsing go down the green hall go all the way then face the chair


 Following instruc2ons: planning go down the green hall go all the way then face the chair


 Following instruc2ons: planning go down the green hall go all the way then face the chair

Following instruc2ons Planning Parsing (no instruc2on) green hall [ go down the hall , face the chair go all the way ]

Previous work Planning Parsing Branavan et al. (2009) Chen and Mooney (2011) Vogel and Jurafsky (2010) Artzi and ZeTlemoyer (2013) Kim and Mooney (2013) Tellex et al. (variously)

Parsing and planning Sequence-to-sequence Structure-to-structure go down the green hall turn le2 go down the green hall

SEQUENCE-TO-SEQUENCE MODEL

Sequence-to-sequence go down the yellow hall turn le2

Sequence-to-sequence go down the yellow hall turn le2 Alignments

Sequence-to-sequence go down the yellow hall turn le2 Alignments Ac2ons 1 2 3

Sequence-to-sequence go down the yellow hall turn le2 Alignments Ac2ons 1 2 3

Sequence-to-sequence go down the yellow hall turn le2 2 Alignments Ac2ons 1 2 3

Sequence-to-sequence go down the yellow hall turn le2 2 Alignments Ac2ons 1 2 3

Sequence-to-sequence go down the yellow hall turn le2 2 1 2 3

Sequence-to-sequence go down the yellow hall turn le2 2 1 2 3

Sequence-to-sequence go down the yellow hall turn le2 2 1 2 3

Sequence-to-sequence go down the yellow hall turn le2 2 1 2 3

Sequence-to-sequence go down the yellow hall turn le2 2 1 2 3

Sequence-to-sequence go down the yellow hall turn le2 ? 2

STRUCTURE-TO-STRUCTURE MODEL

Structure-to-structure go down the yellow hall

Structure-to-structure yellow go down hall the

Structure-to-structure yellow go down hall the

Structure-to-structure the yellow hall go down

Structure-to-structure the yellow hall go down

Structure-to-structure the yellow hall down go

JOINT MODEL

Joint model go down the yellow hall turn le2 2

Joint model go down the yellow hall turn le2 the yellow hall go down

Joint model go down the yellow hall turn le2 the yellow hall down go

Joint model go down the yellow hall turn le2 down the yellow hall go

Joint model ! the yellow θ > ⇣ ⌘ go down ψ φ ( ) + φ ( ) + · · · hall = , , go down

Joint model p (actions , alignment | text; θ ) ∝ ! the yellow X ψ exp hall go down

Joint model go down the yellow hall turn le2 the yellow hall go down

LEARNING / INFERENCE

Learning: coordinate ascent θ , alignments p (actions , alignments | text; θ ) max max alignments using custom alignment dynamic program max θ using L-BFGS

Decoding: coordinate ascent max p (actions , alignments | text; θ ) actions , alignments max alignments using custom alignment dynamic program max actions using a planner

EXPERIMENTS

Three tasks

Puzzle solving clear the two long columns, and then the row [Branavan+ 09]

Puzzle solving clear the two long columns, and then the row [Branavan+ 09]

Puzzle solving Plan execu2on (%) 90 70 60 63 54 30 Branavan et al. No text This work 2009 0

Puzzle solving Task comple2on (%) 90 86 78 60 30 No text This work 0

Maze naviga2on you should see a grey floor to your right and a chair in front of you [MacMahon+ 91]

Maze naviga2on Task comple2on (%) 65 65 63 60 57 57 55 45 Kim + Mooney Chen + Mooney Kim + Mooney Artzi et al. This work 2012 2013 2014 2012 35

Map reading right round the white water [...] but stay quite close ’cause you don’t otherwise you’re going to be in that stone creek [Anderson+ 91]

Map reading Plan execu2on (F 0 ) 60 60 48 45 45 30 15 Vogel+ Andreas+ This work 2010 2013 0

Abla2ons: maze naviga2on Task comple2on (%) 65 60 59 55 52 45 Without Without grounding This work planning graphs 35

Abla2ons: map reading Plan execu2on (F 0 ) 60 60 52 45 45 30 Without 15 Without grounding planning This work graphs 0

Conclusion Structured alignment/decoding gives us best aspects of: ‣ Composi2onal seman2cs (like a parser) ‣ Sequence structure (like a planner)

THANK YOU