SLIDE 1
Evaluating Word Order Recursively over Permutation-Forests Milo s - - PowerPoint PPT Presentation
Evaluating Word Order Recursively over Permutation-Forests Milo s - - PowerPoint PPT Presentation
Evaluating Word Order Recursively over Permutation-Forests Milo s Stanojevi c and Khalil Simaan October 25, 2014 What is wrong with existing metrics related to word order BLEU and many others have local view of word order (n-gram
SLIDE 2
SLIDE 3
What is wrong with existing metrics related to word order
◮ BLEU and many others have local view of word order
(n-gram window) which is not good for long distance reordering.
◮ We need better representation that would allow a global
view – permutations (LRscore, RIBES, FuzzyScore)
SLIDE 4
What is wrong with existing metrics related to word order
◮ BLEU and many others have local view of word order
(n-gram window) which is not good for long distance reordering.
◮ We need better representation that would allow a global
view – permutations (LRscore, RIBES, FuzzyScore)
◮ Problem: not hierarchical and not flexible
SLIDE 5
What is wrong with existing metrics related to word order
◮ BLEU and many others have local view of word order
(n-gram window) which is not good for long distance reordering.
◮ We need better representation that would allow a global
view – permutations (LRscore, RIBES, FuzzyScore)
◮ Problem: not hierarchical and not flexible ◮ Permutation Trees (PETs) might come handy
SLIDE 6
What is wrong with existing metrics related to word order
◮ BLEU and many others have local view of word order
(n-gram window) which is not good for long distance reordering.
◮ We need better representation that would allow a global
view – permutations (LRscore, RIBES, FuzzyScore)
◮ Problem: not hierarchical and not flexible ◮ Permutation Trees (PETs) might come handy ◮ Our metric computes its score in a way similar to PCFG on
these hierarchical structures
SLIDE 7
Recursive metrics
SLIDE 8
Recursive metrics
SLIDE 9
Recursive metrics
SLIDE 10
Recursive metrics
SLIDE 11
Recursive metrics
SLIDE 12
Recursive metrics
SLIDE 13
Recursive metrics
SLIDE 14
PETscore(·) and PEFscore(·)
PETscore(node) =β opScore(node.op)+ (1 − β)
- c∈node.children
PETscore(c)
SLIDE 15
PETscore(·) and PEFscore(·)
PETscore(node) =β opScore(node.op)+ (1 − β)
- c∈node.children
PETscore(c) But, there might be (exponentially) many PETs for a single permutation! PEFscore(π) =
- t∈PEF(π) PETscore(t)
#PETs Can be efficiently computed with a version of Inside algorithm.
SLIDE 16
Results into English (scaled Kendall sent level)
SLIDE 17
Results out of English (scaled Kendall sent level)
SLIDE 18
Conclusion
◮ Consider all factorizations (PEF)
SLIDE 19
Conclusion
◮ Consider all factorizations (PEF) ◮ Do it hierarchically
SLIDE 20
Conclusion
◮ Consider all factorizations (PEF) ◮ Do it hierarchically ◮ Metric is available online together with BEER
SLIDE 21
Conclusion
◮ Consider all factorizations (PEF) ◮ Do it hierarchically ◮ Metric is available online together with BEER ◮ Come to see the poster
SLIDE 22