SLIDE 1
Alignment-Based Composi2onal Seman2cs for Instruc2on Following - - PowerPoint PPT Presentation
Alignment-Based Composi2onal Seman2cs for Instruc2on Following - - PowerPoint PPT Presentation
Alignment-Based Composi2onal Seman2cs for Instruc2on Following Jacob Andreas and Dan Klein UC Berkeley Following instruc2ons go down the green hall go all the way then face the chair Following instruc2ons go down the green hall
SLIDE 2
SLIDE 3
Following instruc2ons go down the green hall go all the way then face the chair
SLIDE 4
Following instruc2ons: parsing go down the green hall go all the way then face the chair
SLIDE 5
Following instruc2ons: parsing go down the green hall go all the way then face the chair
SLIDE 6
Following instruc2ons: planning go down the green hall go all the way then face the chair
SLIDE 7
Following instruc2ons: planning go down the green hall go all the way then face the chair
SLIDE 8
Following instruc2ons Planning Parsing
[ go down the hall, go all the way ] green hall face the chair (no instruc2on)
SLIDE 9
Previous work Planning Parsing
Branavan et al. (2009) Vogel and Jurafsky (2010) Chen and Mooney (2011) Artzi and ZeTlemoyer (2013) Kim and Mooney (2013) Tellex et al. (variously)
SLIDE 10
Parsing and planning Sequence-to-sequence Structure-to-structure
go down the green hall turn le2 go down the green hall
SLIDE 11
SEQUENCE-TO-SEQUENCE MODEL
SLIDE 12
go down the yellow hall turn le2
Sequence-to-sequence
SLIDE 13
go down the yellow hall turn le2
Sequence-to-sequence
Alignments
SLIDE 14
go down the yellow hall turn le2
Sequence-to-sequence
1 2 3 Ac2ons Alignments
SLIDE 15
go down the yellow hall turn le2
Sequence-to-sequence
1 2 3 Ac2ons Alignments
SLIDE 16
go down the yellow hall turn le2
Sequence-to-sequence
1 2 3 Ac2ons Alignments
2
SLIDE 17
go down the yellow hall turn le2
Sequence-to-sequence
1 2 3 Ac2ons Alignments
2
SLIDE 18
go down the yellow hall turn le2
Sequence-to-sequence
1 2 3
2
SLIDE 19
go down the yellow hall turn le2
2
Sequence-to-sequence
1 2 3
SLIDE 20
go down the yellow hall turn le2
2
Sequence-to-sequence
1 2 3
SLIDE 21
go down the yellow hall turn le2
2
Sequence-to-sequence
1 2 3
SLIDE 22
Sequence-to-sequence
go down the yellow hall turn le2
2
1 2 3
SLIDE 23
go down the yellow hall turn le2
Sequence-to-sequence
2
?
SLIDE 24
STRUCTURE-TO-STRUCTURE MODEL
SLIDE 25
Structure-to-structure
go down the yellow hall
SLIDE 26
Structure-to-structure
go down hall the yellow
SLIDE 27
Structure-to-structure
go down hall the yellow
SLIDE 28
Structure-to-structure
go down the yellow hall
SLIDE 29
Structure-to-structure
go down the yellow hall
SLIDE 30
Structure-to-structure
go down hall the yellow
SLIDE 31
JOINT MODEL
SLIDE 32
2
go down the yellow hall turn le2
Joint model
SLIDE 33
go down the yellow hall turn le2
Joint model
go down the yellow hall
SLIDE 34
go down the yellow hall turn le2
Joint model
go down hall the yellow
SLIDE 35
go down the yellow hall turn le2
Joint model
go down the yellow hall
SLIDE 36
Joint model
ψ !
θ>⇣
⌘
go
φ( , )
down
+ φ( , )
=
+ · · ·
go down the yellow hall
SLIDE 37
Joint model
ψ !
exp X
p(actions, alignment | text; θ) ∝
go down the yellow hall
SLIDE 38
go down the yellow hall turn le2
Joint model
go down the yellow hall
SLIDE 39
LEARNING / INFERENCE
SLIDE 40
Learning: coordinate ascent
max
θ,alignments p(actions, alignments | text; θ)
max
θ
max
alignments
using L-BFGS using custom alignment dynamic program
SLIDE 41
Decoding: coordinate ascent
max
alignments
using a planner using custom alignment dynamic program
max
actions
max
actions, alignments
p(actions, alignments | text; θ)
SLIDE 42
EXPERIMENTS
SLIDE 43
Three tasks
SLIDE 44
Puzzle solving
clear the two long columns, and then the row
[Branavan+ 09]
SLIDE 45
Puzzle solving
clear the two long columns, and then the row
[Branavan+ 09]
SLIDE 46
Puzzle solving
30 60 90
70 63 54
No text
Branavan et al.
2009 This work
Plan execu2on (%)
SLIDE 47
Puzzle solving
30 60 90
86 78
No text This work
Task comple2on (%)
SLIDE 48
Maze naviga2on
you should see a grey floor to your right and a chair in front of you
[MacMahon+ 91]
SLIDE 49
Maze naviga2on
35 45 55 65
65 63 60 57 57
Kim + Mooney
2012
Chen + Mooney
2012 This work
Kim + Mooney
2013
Artzi et al.
2014
Task comple2on (%)
SLIDE 50
Map reading
right round the white water [...] but stay quite close ’cause you don’t
- therwise you’re going to
be in that stone creek
[Anderson+ 91]
SLIDE 51
Map reading
15 30 45 60
60 45 48
Vogel+ 2010 Andreas+ 2013 This work
Plan execu2on (F0)
SLIDE 52
Abla2ons: maze naviga2on
35 45 55 65
60 52 59
Without grounding graphs
Task comple2on (%)
This work Without planning
SLIDE 53
Abla2ons: map reading
15 30 45 60
60 52 45
Without grounding graphs This work
Plan execu2on (F0)
Without planning
SLIDE 54
Conclusion
- Composi2onal seman2cs (like a parser)
- Sequence structure (like a planner)
Structured alignment/decoding gives us best aspects of:
SLIDE 55