!"#$%&'%()#*&$+%,'-.#-/0%1(,23% - - PDF document

▶
0 1 23
SMART_READER_LITE
LIVE PREVIEW

!"#$%&'%()#*&$+%,'-.#-/0%1(,23% - - PDF document

Mar 28, 2024 575 likes •1.04k views

Marco Turchi turchi@fbk.eu Statistical Machine Translation - ICT Doctoral School 19 March 2013 Quality Estimation Turchi, FBK 1 !"#$%&'%()#*&$+%,'-.#-/0%1(,23% 4%5&$%/6%"&'$/7+%

slide-1
SLIDE 1

Marco Turchi turchi@fbk.eu

Statistical Machine Translation - ICT Doctoral School – 19 March 2013

Quality Estimation – Turchi, FBK 1 Quality Estimation – Turchi, FBK

  • !"#$%&'%()#*&$+%,'-.#-/0%1(,23%

– 4%5&$%/6%"&'$/7+% – ()#*&$+%809&:#$/7'% – ()#*&$+%;:/7<'% – =<#70&0>%4*>/7&$".'% – ?#$#'<$% – ,@#*)#-/0%A<$7&:'% – BC<0%8'')<'%

  • ()#*&$+%,'-.#-/0%#$D%

– !/79%=<@<*% – ?/:).<0$%=<@<*%

  • E/F%:#0%F<%)'<%(,%&0%7<#*%#CC*&:#-/0'3%

2

slide-2
SLIDE 2

Quality Estimation – Turchi, FBK

  • 4)$/.#-:%,@#*)#-/0%/6%AG%/)$C)$%

%;/)7:<% =#0>H% G<'$%;<$% G7#0'*#$<9% G<'$%;<$%% ;AG%

3 Quality Estimation – Turchi, FBK

  • I%./7<%9&J:)*$%&6%$"<7<%#7<%./7<%;AG%'+'$<.'KKK%

%;/)7:<% =#0>H% G<'$%;<$% ;AG% ;AG% ;AG% ;AG% ;AG% G<'$%;<$'%

4

slide-3
SLIDE 3

Quality Estimation – Turchi, FBK

  • E).#0%C7<'<0:<%

%;/)7:<% =#0>H% G<'$%;<$% G7#0'*#$<9% G<'$%;<$%% ;AG%

  • I%5)$%&$%&'D%
  • ')5L<:-@<%
  • -.<%:/0').&0>%
  • 9&J:)*$%
  • 0/$%7<C*&:#5*<%

5 Quality Estimation – Turchi, FBK

%;/)7:<% =#0>H% G<'$%;<$% G<'$%;<$% M<6<7<0:<% G<'$%;<$%% G7#0'*#$<9% G<'$%;<$%% ;AG% N=,O% !,M% G,M% A<$</7% P8;G% I% ;:/7<%

  • M<6<7<0:<%G<'$%'<$D%").#0%$7#0'*#-/0%/6%

$"<%'/)7:<%*#0>)#><%$<'$%'<$%&0$/%$"<% $#7><$%*#0>)#><H%

6

slide-4
SLIDE 4

Quality Estimation – Turchi, FBK

  • =#7><%#./)0$%/6D%

– %C#7#**<*%$<'$%'<$'%&0%9&Q<7<0$%*#0>)#><'D%

  • !/7R'"/C%/6%A#:"&0<%G7#0'*#-/0%1!AG2%STTUV%IV%STWXV%

,)7/C#7*V%4Y)&'%Z/..)0&$#&7<V%P8;G%:#.C#&>0V%I%

– 4)$/.#-:%;:/7&0>%A<$"/9'%1N*<)V%G,MV%!,MV%A<$</72D%

  • */F%:/'$%1F7$%").#0%<@#*)#-/02%
  • /5L<:-@<%1)05&#'<92%
  • &06/7.#-@<%16/7%'+'$<.%9<@<*/C<7'2D%$/%C7/[*<%'+'$<.%5<"#@&/7%
  • 9&':7&.&0#-@<D%$/%$<**%&6%#09%F"<7<%&.C7/@<.<0$'%#7<%
  • <Q<:-@<%#09%7<C*&:#5*<D%$/%5<%:/.C)$<9%Y)&:R*+%#09%/\<0%

7 Quality Estimation – Turchi, FBK

  • ]<76<:$%'<^0>%&0%$"<%:/0$7/**<9%':<0#7&/%1*#5%$<'$'V%

<@#*)#-/0%:#.C#&>0'V%I2%F"<7<%&$%&'%7<Y)&7<9%$/%:/.C#7<% '<@<7#*%'+'$<.'%/0%#%>&@<0%$<'$%'<$H%

  • 5)$%I%

?

8

slide-5
SLIDE 5

Quality Estimation – Turchi, FBK

  • In real applications, several questions :

– Is the translated text good enough to be published? – Can we trust it? – Can a reader get the gist? – Is it worst to post-edit it?

  • Quality Estimation (QE) can help us to find the

right answers!!

9

%;/)7:<% ;<0$<0:<% G7#0'*#$<9% ;<0$<0:<% ;AG%

Quality Estimation – Turchi, FBK

  • ()#*&$+%,'-.#-/0%1(,2D%

– .<$7&:'%$"#$%C7/@&9<%#0%<'-.#$<%/0%$"<%Y)#*&$+%/6% )0'<<0%$7#0'*#$<9%$<_$'%F&$"/)$%$"<%7<6<7<0:<% $7#0'*#-/0'%% – #%')C<7@&'<9%*<#70&0>%$#'RV%F"<7<%#%.#:"&0<%$7&<'% $/%./9<*%"/F%").#0'%L)9><%$7#0'*#$<9%$<_$'% 1*<#70&0>%67/.%").#0%*#5<*<9%9#$#2%%

10

slide-6
SLIDE 6

Quality Estimation – Turchi, FBK

%;/)7:<% =#0>H% G<'$%;<$% G7#0'*#$<9% G<'$%;<$%% ;AG% ()#*&$+% ,'-.#$/7% (,% ]7<9&:-/0`% `%':/7<%#$%'<0$<0:<%*<@<*% =#5<*<9% ?#$#%

11 Quality Estimation – Turchi, FBK

  • Automatic Evaluation:

– require parallel test sets – fundamental in controlled experiments – compare different system performances

  • Quality Estimation:

– does not require parallel data, but labeled data – application oriented

  • Both very complex tasks

12

slide-7
SLIDE 7

Quality Estimation – Turchi, FBK

  • Example 1:
  • ;/)7:<D%

– a80%Y)<'$#%'&$)#b&/0<V%c%C/''&5&*<%:7<#7<%)0% '#*@#$#>>&/d%#*%$<7.&0<V%&*%'#*@#$#>>&/%c%&09&C<09<0$<% 9#*%@/*).<%9&%C7/9)b&/0<He%

  • G#7><$D%

– a80%$"&'%'&$)#-/0V%+/)%:#0%:7<#$<%#%5#:R)CV%#\<7%$"<% 5#:R)C%.#+%5<%&09<C<09<0$%/6%$"<%@/*).<%/6% C7/9):-/0Hf%

13

  • Can a reader get the gist?
  • Is the translated text good enough to be

published?

Quality Estimation – Turchi, FBK

  • Example 2:
  • ;/)7:<D%

– a80%Y)<'$#%'&$)#b&/0<V%c%C/''&5&*<%:7<#7<%)0% '#*@#$#>>&/d%#*%$<7.&0<V%&*%'#*@#$#>>&/%c%&09&C<09<0$<% 9#*%@/*).<%9&%C7/9)b&/0<He%

  • G#7><$D%

– a80%$"&'%'&$)#-/0V%+/)%:7<#$<%#%'#*@#$#>>&/%#09%#\<7%&$% :/.C*<$<'V%$"<%'#*@#$#>>&/%&'%&09<C<09<0$%/6%$"<% C7/9):-/0%@/*).<Hf%

14

  • Can a reader get the gist?
  • Is the translated text good enough to be

published?

slide-8
SLIDE 8

Quality Estimation – Turchi, FBK

  • Example 3:
  • ;/)7:<D%

– a80%Y)<'$#%'&$)#b&/0<V%c%C/''&5&*<%:7<#7<%)0% '#*@#$#>>&/d%#*%$<7.&0<V%&*%'#*@#$#>>&/%c%&09&C<09<0$<% 9#*%@/*).<%9&%C7/9)b&/0<He%

  • G#7><$D%

– a80%$"&'%'&$)#-/0V%+/)%:#0%:7<#$<%#%5#:R)CV%#\<7%$"<% 5#:R)C%.#+%5<%&09<C<09<0$%/6%$"<%@/*).<%/6% C7/9):-/0Hf%

  • 1(Bad)

1 (Good)

15 Quality Estimation – Turchi, FBK

  • Example 4:
  • ;/)7:<D%

– a80%Y)<'$#%'&$)#b&/0<V%c%C/''&5&*<%:7<#7<%)0% '#*@#$#>>&/d%#*%$<7.&0<V%&*%'#*@#$#>>&/%c%&09&C<09<0$<% 9#*%@/*).<%9&%C7/9)b&/0<He%

  • G#7><$D%

– a80%$"&'%'&$)#-/0V%+/)%:#0%:7<#$<%#%5#:R)CV%#\<7%$"<% 5#:R)C%.#+%5<%&09<C<09<0$%/6%$"<%@/*).<%/6% C7/9):-/0Hf% 1(Bad) 2(Fair) 3(Partially Good) 4(Good)

16

slide-9
SLIDE 9

Quality Estimation – Turchi, FBK

  • ;/)7:<D%

– a80%Y)<'$#%'&$)#b&/0<V%c%C/''&5&*<%:7<#7<%)0% '#*@#$#>>&/d%#*%$<7.&0<V%&*%'#*@#$#>>&/%c%&09&C<09<0$<% 9#*%@/*).<%9&%C7/9)b&/0<He%

  • G#7><$D%

– a80%$"&'%'&$)#-/0V%+/)%:#0%:7<#$<%#%5#:R)CV%#\<7%$"<% 5#:R)C%.#+%5<%&09<C<09<0$%/6%$"<%@/*).<%/6% C7/9):-/0Hf%

  • ]/'$g,9&$%;<0$<0:<D%

– a80%$"&'%'&$)#-/0V%+/)%:7<#$<%#%5#:R)C%#09%#\<7%&$% :/.C*<$<'V%$"<%5#:R)C%&'%&09<C<09<0$%/6%$"<% C7/9):-/0%@/*).<Hf%

17

  • !"#$%:#0%(,%C7<9&:$3%

– 7<>7<''&/0%C7/5*<.D%#%7<#*%0).5<7%Y)#*&$+%':/7<%1<H>H% hTHHUi2% – :*#''&[:#-/0%C7/5*<.D%#%:#$<>/7+%*#5<*%1<H>H%5#9%/7%>//92 % – 7#0R&0>%C7/5*<.D%$"<%7#0R%/6%#%'<$%/6%'<0$<0:<'%1<H>H% 'UV'XV'jV%I2%

  • !"<7<%:#0%(,%5<%)'<93D%

– '<0$<0:<%*<@<*% – F/79%*<@<*% – 9/:).<0$%*<@<*%%%

Quality Estimation – Turchi, FBK 18

slide-10
SLIDE 10

Quality Estimation – Turchi, FBK

  • !"#$%&'%()#*&$+%,'-.#-/0%1(,23%

– 4%5&$%/6%"&'$/7+% – ()#*&$+%809&:#$/7'% – ()#*&$+%;:/7<'% – =<#70&0>%4*>/7&$".'% – ?#$#'<$% – ,@#*)#-/0%A<$7&:'% – BC<0%8'')<'%

  • ()#*&$+%,'-.#-/0%#$D%

– !/79%=<@<*% – ?/:).<0$%=<@<*%

  • E/F%:#0%F<%)'<%(,%&0%7<#*%#CC*&:#-/0'3%

19 Quality Estimation – Turchi, FBK

  • ()#*&$+%,'-.#-/0%&'%#0%/*9k0<F%$/C&:H%
  • lEO%!/7R'"/C%STTX%mlEOVSTTXnD%

– ]7<9&:$%N=,OV%G,M%#09%!,M% – Bad%AG%/)$C)$%! hard to beat the baselines – No real use in applications

20

slide-11
SLIDE 11

Quality Estimation – Turchi, FBK

  • Specia et al, 2009 [Spe,2009]:

– Better MT output – More attention in the private sector – Predict Post-Editing (PE) Effort

  • human effort to transform the MT output into a good text

(Post-editing time, human scores, % errors to fix)

– First dataset freely available

21 Quality Estimation – Turchi, FBK

  • Specia et al, 2010 [Spe,2010]:

– New interpretation of QE:

– Predict the MT evaluation: – Select the best translation from multiple MT-systems – Filtering out bad translations

– Regression vs Binary Classification vs Multi-label Classification

22

slide-12
SLIDE 12

Quality Estimation – Turchi, FBK

  • !/7R'"/C%/0%AGV%STWSV%'"#7<9%$#'R%()#*&$+%,'-.#-/0%mZ#*VWSn%%

– Training/test data with human annotation (1..5) – 11 participants – Scoring and ranking tasks – New quality indicators (features) – Tested a large number of machine learning techniques

  • Workshop on MT, 2013, shared task Quality Estimation…

– coming soon

23 Quality Estimation – Turchi, FBK

  • !"#$%&'%()#*&$+%,'-.#-/0%1(,23%

– 4%5&$%/6%"&'$/7+% – ()#*&$+%809&:#$/7'% – ()#*&$+%;:/7<'% – =<#70&0>%4*>/7&$".'% – ?#$#'<$% – ,@#*)#-/0%A<$7&:'% – BC<0%8'')<'%

  • ()#*&$+%,'-.#-/0%#$D%

– !/79%=<@<*% – ?/:).<0$%=<@<*%

  • E/F%:#0%F<%)'<%(,%&0%7<#*%#CC*&:#-/0'3%

24

slide-13
SLIDE 13

Quality Estimation – Turchi, FBK

%;/)7:<% =#0>H% G<'$%;<$% G7#0'*#$<9% G<'$%;<$%% ;AG% ()#*&$+% ,'-.#$/7% (,% ]7<9&:-/0`% `%':/7<%#$%'<0$<0:<%*<@<*% =#5<*<9% ?#$#%

25

  • 80%$"<%*<#70&0>%67#.<F/7RD%

– ()#*&$+%809&:#$/7'D%%

  • :#$:"%Y)#*&$+%&06/7.#-/0%&0%'/)7:<%#09%$#7><$%'<0$<0:<'%

– ()#*&$+%;:/7<D%%

  • .<$7&:'%$/%<_C7<''%Y)#*&$+%/6%$"<%$#7><$%'<0$<0:<%

– =<#70&0>%4*>/7&$".D%%

  • *<#70'%"/F%$/%C<76/7.%#%:/.C*<_%$#'R%1./'$%)'<9D%;oAV%]=;V%

AU]2%

– =#5<*<9%?#$#D%").#0%#00/$#$<9%'<0$<0:<%C#&7'%

Quality Estimation – Turchi, FBK 26

slide-14
SLIDE 14
  • ()#*&$+%809&:#$/7'%:#0%5<%<_$7#:$<9%67/.D%

– ;/)7:<%'<0$<0:<%1Z/.C*<_&$+%&09&:#$/7'2% – G#7><$%'<0$<0:<%1p*)<0:+%&09&:#$/7'2% – ;AG%'+'$<.%9)7&0>%$"<%$7#0'*#-/0%C7/:<''%1Z/0[9<0:<% &09&:#$/7'2% – ;/)7:<%#09%G#7><$%'<0$<0:<'%149<Y)#:+%#09%/$"<7% &09&:#$/7'2%

Quality Estimation – Turchi, FBK

%;/)7:<% ;<0$<0:<% G7#0'*#$<9% ;<0$<0:<% ;AG%

27

  • Z/.C*<_&$+%809&:#$/7'D%

– .<#')7<%$"<%*<@<*%/6%$7#0'*#-/0%9&J:)*$+%/6%$"<%'/)7:<% '<0$<0:<% – '<0$<0:<'%$"#$%#7<%:/.C*<_%#$%$"<%'+0$#:-:#*V%'<.#0-:V% 9&':)7'&@<%/7%C7#>.#-:%*<@<*'%#7<%"#79<7%$/%$7#0'*#$<%%

Quality Estimation – Turchi, FBK

%;/)7:<% ;<0$<0:<% G7#0'*#$<9% ;<0$<0:<% ;AG%

28

slide-15
SLIDE 15
  • <H>H%

– '/)7:<%'<0$<0:<%*<0>$"% – 0g>7#.%*#0>)#><%./9<*%C7/5#5&*&$+% – '/)7:<%'<0$<0:<%$+C<%$/R<0%7#-/0%1<H>H%q0/)0'kq$/R<0'2% – #@>H%q%/6%$7#0'*#-/0'%C<7%'/)7:<%F/79%&0%$"<%'<0$<0:<%1#'%>&@<0% 5+%C7/5#5&*&'-:%9&:-/0#7&<'2% – r%/6%q%/6%:/0$<0$k0/0g:/0$<0$%F/79'H%

Quality Estimation – Turchi, FBK

%;/)7:<% ;<0$<0:<% G7#0'*#$<9% ;<0$<0:<% ;AG%

29

  • p*)<0:+%809&:#$/7'D%

– .<#')7<%$"<%*<@<*%/6%0#$)7#*0<''%/6%$"<%$#7><$%%'<0$<0:<%&0% $"<%$#7><$%*#0>)#><% – $#7><$%'<0$<0:<%$"#$%9/<'%0/$%:/06/7.%$/%$"<%$#7><$% *#0>)#><%%&0%$<7.'%/6%>7#..#7V%#09%F&$"%&0#CC7/C7&#$<% *<_&:#*%:"/&:<'%6/7%$"<%:/0$<_$%#09%$"<%><07<%/6%$"<%$<_$%:#0 % 5<%<@&9<0:<%/6%#%5#9%$7#0'*#-/0%%

Quality Estimation – Turchi, FBK

%;/)7:<% ;<0$<0:<% G7#0'*#$<9% ;<0$<0:<% ;AG%

30

slide-16
SLIDE 16
  • <H>H%

– $#7><$%'<0$<0:<%*<0>$"% – 0g>7#.%*#0>)#><%./9<*%C7/5#5&*&$+% – $#7><$%'<0$<0:<%$+C<%$/R<0%7#-/0%1<H>H%q0/)0'k q$/R<0'2% – ]B;g$#>%$#7><$%*#0>)#><%./9<*%

Quality Estimation – Turchi, FBK

%;/)7:<% ;<0$<0:<% G7#0'*#$<9% ;<0$<0:<% ;AG%

31

  • Z/0[9<0:<%809&:#$/7'D%

– .<#')7<%$"<%*<@<*%/6%:/0[9<0:<%/6%$"<%;AG%'+'$<.%&0% C7/9):&0>%$"<%$#7><$%%'<0$<0:<% – '<0$<0:<'%6/7%F"&:"%$"<%$7#0'*#-/0%C7/:<''%&'%:/.C*<_%#7<% ./7<%*&R<*+%$/%5<%5#9%$7#0'*#-/0'%

Quality Estimation – Turchi, FBK

%;/)7:<% ;<0$<0:<% G7#0'*#$<9% ;<0$<0:<% ;AG%

32

slide-17
SLIDE 17
  • <H>H%

– *<0>$"%P%/6%$"<%P%5<'$%*&'$% – q%/6%C7)0<9%"+C/$"<'&'% – */>g*&R<*&"//9%':/7<% – #@>H%<9&$g9&'$#0:<%/6%$"<%Wg5<'$%67/.%$"<%[7'$%Rg5<'$'%

Quality Estimation – Turchi, FBK

%;/)7:<% ;<0$<0:<% G7#0'*#$<9% ;<0$<0:<% ;AG%

33

  • 49<Y)#:+%809&:#$/7'D%

– .<#')7<%$"<%*<@<*%/6%&06/7.#-/0%:/0@<+<9%&0$/%$"<%$#7><$% '<0$<0:<% – '/)7:<%#09%$#7><$%'<0$<0:<'%'"/)*9%:/0@<+%$"<%'#.<% .<#0&0>H%;.#**%#./)0$%/6%$"<%'/)7:<%.<#0&0>%/6%$"<%$#7><$% '<0$<0:<%:#0%<@&9<0:<%/6%#%5#9%$7#0'*#-/0%

Quality Estimation – Turchi, FBK

%;/)7:<% ;<0$<0:<% G7#0'*#$<9% ;<0$<0:<% ;AG%

34

slide-18
SLIDE 18
  • <H>H%
  • r%/6%#*&>0<9%F/79'%&0%$"<%'/)7:<%#09%$#7><$%'<0$<0:<'%
  • q%/6%#*&>0.<0$'%5<$F<<0%$"<%'/)7:<%#09%$#7><$%'<0$<0:<'%
  • 6/7%<#:"%$#7><$%F/79V%q%-.<%#*&>0<9%$/%$"#$%'/)7:<%F/79%&0%$"<%

'/)7:<%'<0$<0:<%9&@&9<9%5+%q%/6%C/$<0-#*%#*&>0.<0$'%&0%$"<%GA%

  • 6/7%<#:"%]/;%$#>%CV%r%0%/6%#*&>0<9%F/79'%&0%$"<%'/)7:<%/7%$#7><$%

'<0$<0:<'%$#>><9%F&$"%C%

Quality Estimation – Turchi, FBK

%;/)7:<% ;<0$<0:<% G7#0'*#$<9% ;<0$<0:<% ;AG%

35

  • !"&:"%&'%$"<%5<'$%>7/)C3%%E#79%$/%'#+%
  • !"&:"%#7<%$"<%5<'$%6<#$)7<'3%E#79%$/%'#+%

– '/.<%6<#$)7<'%C<76/7.%5<s<7%$"#0%/$"<7'V%5)$%$"<7<%#7<% 6<F%9#$#'<$'%$/%$<'$%$"<.% – *<''/0%*<#70<9%#$%!AGV%(,%$#'RD%

  • :/0[9<0:<D%./9<*%:/.C/0<0$'%67/.%;AG%9<:/9<7%
  • C'<)9/g7<6<7<0:<D%#>7<<.<0$%5<$F<<0%S%;AG%
  • '+'$<.'%6)bb+g.#$:"%*&R<D%'/)7:<%1#09%$#7><$2%'&.&*#7&$+%F&$"%;AG%$7#&0&0>%

:/7C)'%1=AV%<$:2%

  • P<F%7<'<#7:"%$7<09'D%6<#$)7<%'<*<:-/0%#09%&9<0-[:#-/0%/6%

$"<%:/0$7&5)$<%/6%<#:"%>7/)C%$/%$"<%[0#*%':/7<%

Quality Estimation – Turchi, FBK 36

slide-19
SLIDE 19

Quality Estimation – Turchi, FBK

  • !"#$%&'%()#*&$+%,'-.#-/0%1(,23%

– 4%5&$%/6%"&'$/7+% – ()#*&$+%809&:#$/7'% – ()#*&$+%;:/7<'% – =<#70&0>%4*>/7&$".'% – ?#$#'<$% – ,@#*)#-/0%A<$7&:'% – BC<0%8'')<'%

  • ()#*&$+%,'-.#-/0%#$D%

– !/79%=<@<*% – ?/:).<0$%=<@<*%

  • E/F%:#0%F<%)'<%(,%&0%7<#*%#CC*&:#-/0'3%

37

  • !"&:"%&'%#%>//9%&09&:#$/7%/6%$7#0'*#-/0%Y)#*&$+%$/%

C7<9&:$3%

  • 40%&09&:#$/7%/6%$7#0'*#-/0%Y)#*&$+%'"/)*9%$#R<%&0$/%

#::/)0$D%

– :/77<:$0<''%#09%)-*&$+%/6%$"<%$7#0'*#-/0% – :/>0&-@<%<Q/7$%0<<9<9%5+%%").#0%$/%:/77<:$%$"<% $7#0'*#-/0%

  • 4**%$"<'<%#'C<:$'%:#0%5<%')..#7&b<9%&0%$"<D%

– ]/'$g,9&-0>%,Q/7$%

Quality Estimation – Turchi, FBK 38

slide-20
SLIDE 20
  • !"#$%&'%]/'$g,9&-0>3%1`2%

– 4%C7/:<''%/6%./9&[:#-/0%7#$"<7%$"#0%7<@&'&/0H% 1=/t<7g=#)7&#0%WuvU2% – G"<%a$<7.%)'<9%6/7%$"<%:/77<:-/0%/6%.#:"&0<% $7#0'*#-/0%/)$C)$%5+%").#0%*&0>)&'$'k<9&$/7'f%1o<#*<% #09%!#+%Wuuw2% – M<C#&7&0>%$<_$'%1x7&0>'V%STTW2% – aI$"<%C7/:<''%/6%&.C7/@&0>%#%.#:"&0<g><0<7#$<9% $7#0'*#-/0%F&$"%#%.&0&.).%/6%.#0)#*%*#5/7f%1G4O;% 7<C/7$V%STWT2%

Quality Estimation – Turchi, FBK

`%67/.%G)$/7&#*%#$%4AG4%STWTH%M<6<7<0:<'%#@#&*#5*<%&0%mByN7VWTn%

39

  • ]/'$g,9&-0>%,Q/7$%&'%

– $"<%<Q/7$%.#9<%5+%#%C/'$g<9&$/7%$/%.#0)#**+%&.C7/@<%#% .#:"&0<%><0<7#$<9%$7#0'*#-/0%

  • ]/'$g,9&-0>%,Q/7$%&'%0/$%#%':/7<kY)#0-$+%
  • P<<9%':/7<'%#5*<%$/%.<#')7<%$"<%C/'$g<9&-0>%<Q/7$D%

– ;:/7&0>%':"<.<% – ]/'$g,9&-0>%-.<%1'<:/09'%/7%'<:/09'%C<7%F/79'2%%% – EG,M% – P).5<7%/6%R<+'$7/R<'% – I%

Quality Estimation – Turchi, FBK 40

slide-21
SLIDE 21
  • ;:/7&0>%;:"<.#D%

– #75&$7#7+%:"/&:<%/6%$"<%*<@<*'%/6%Y)#*&$+%%

  • #%'<0$<0:<%:#0%5<%*#5<*<9%#'D%%

% W%z%7<Y)&7<'%:/.C*<$<%7<$7#0'*#-/0d% % S%z%7<Y)&7<'%'/.<%7<$7#0'*#-/0d%% % X%z%@<7+%*&s*<%C/'$%<9&-0>%0<<9<9d%% % {%z%[$%6/7%C)7C/'<% %

– *#5<*&0>%7<Y)&7<'%$"<%").#0%&0$<7@<0-/0% – #%C7<:&'<%.<#')7<%% – ')5L<:-@<k<_C<0'&@<k-.<%:/0').&0>%$#'R%

Quality Estimation – Turchi, FBK 41

  • !/7R'"/C%/0%;$#-'-:#*%A#:"&0<%G7#0'*#-/0%;:/7&0>%;:"<.#D%

– W%g%G"<%AG%/)$C)$%&'%&0:/.C7<"<0'&5*<V%F&$"%*&s*<%/7%0/%&06/7.#-/0% $7#0'6<77<9%#::)7#$<*+H%8$%:#00/$%5<%<9&$<9V%0<<9'%$/%5<%$7#0'*#$<9%67/.% ':7#$:"H% % – S%g%45/)$%UTr%gwTr%/6%$"<%AG%/)$C)$%0<<9'%$/%5<%<9&$<9H%8$%7<Y)&7<'%#% '&>0&[:#0$%<9&-0>%<Q/7$%&0%/79<7%$/%7<#:"%C)5*&'"#5*<%*<@<*H%% – X%g%45/)$%SUgUTr%/6%$"<%AG%/)$C)$%0<<9'%$/%5<%<9&$<9H%8$%:/0$#&0'% 9&Q<7<0$%<77/7'%#09%.&'$7#0'*#-/0'%$"#$%0<<9%$/%5<%:/77<:$<9H% % % – {%g%45/)$%WTgSUr%/6%$"<%AG%/)$C)$%0<<9'%$/%5<%<9&$<9H%8$%&'%><0<7#**+%:*<#7% #09%&0$<**&>&5*<H%% % – U%g%G"<%AG%/)$C)$%&'%C<76<:$*+%:*<#7%#09%&0$<**&>&5*<H%%8$%&'%0/$%0<:<''#7&*+%#% C<76<:$%$7#0'*#-/0V%5)$%7<Y)&7<'%*&s*<%$/%0/%<9&-0>H% %

Quality Estimation – Turchi, FBK 42

slide-22
SLIDE 22

Quality Estimation – Turchi, FBK 43

  • ]/'$g,9&-0>%G&.<%

– -.<%1'<:/09'2%$/%C/'$g<9&$%#%'<0$<0:<% – 0/7.#*&b<9%@<7'&/0%&0%'<:/09'%C<7%F/79% – *&s*<%-.<%&.C*&<'%>//9%$7#0'*#-/0% – *#7><%-.<%&.C*&<'%5#9%$7#0'*#-/0%

  • O')#**+%&0:*)9<'D%

– 7<#9&0>%-.<% – '<#7:"%/6%&06/7.#-/0%/0%<_$<70#*%7<'/)7:<'% – $+C&0>%-.<% – <_$7#%-.<%6/7%'<:/09#7+%#:-@&$+%10/&'<2H%<H>H%:/77<:-/0%/6%"&'%F7&s<0% $<_$%%

  • E&>"%@#7&#5&*&$+%#:7/''%'<0$<0:<'%#09%$7#0'*#$/7'%

Quality Estimation – Turchi, FBK 44

slide-23
SLIDE 23

Quality Estimation – Turchi, FBK

  • #%>//9k5#9%$7#0'*#-/0%%!%0/$%>//9k5#9%6/7%#**%$7#0'*#$/7'%1'<<%%v2%
  • :/0'&9<7'%9&Q<7<0$%<Q/7$'%&0%$7#0'*#-0>%9&Q<7<0$%F/79'%
  • C7<'<0:<%/6%/)$*&<7'%

45

  • ]/'$g,9&-0>%G&.<%9<:#+'%<_C/0<0-#**+%6/7%'<0$<0:<'%

5<*/0>&0>%$/%$"<%'#.<%9/:).<0$%

– ]/'$g<9&$/7'%*<#70%9/:).<0$%$<7.&0/*/>+V%'$+*<V% <_C7<''&/0'V%I%9)7&0>%$"<%$7#0'*#-/0%C7/:<''%

Quality Estimation – Turchi, FBK 46

slide-24
SLIDE 24
  • G,M%1`2%
  • E).#0%$#7><$<9%G,M%&'%$"<%'$#09#79%<9&$%9&'$#0:<%5<$F<<0%

$"<%/7&>&0#*%.#:"&0<%$7#0'*#-/0%#09%&$'%.&0&.#**+%C/'$g <9&$<9%@<7'&/0%

– <9&$'D%&0'<7-/0V%9<*<-/0V%')5'-$)-/0V%'"&\%% – #$%'<0$<0:<%*<@<*%&0%$"<%C/'$g<9&-0>%':<0#7&/V%EG,M%&'% 0/7.#*&b<9%5+%&$'%*<0>$"%

Quality Estimation – Turchi, FBK

`%'<<%*<''/0%5+%A#)7/%Z<s/*/%

HTER = #edits # words_ postedited _version

47

TER = #edits avg_#of _ reference_ words

  • 40%<_#.C*<%

Quality Estimation – Turchi, FBK 48

slide-25
SLIDE 25

Quality Estimation – Turchi, FBK 49 Quality Estimation – Turchi, FBK

  • EG,MD%

– *<''%@#7&#0:<%#:7/''%9&Q<7<0$%C/'$g<9&$/7'D% – %#%>//9k5#9%$7#0'*#-/0%!%>//9k5#9%6/7%#**%$7#0'*#$/7'% – 8>0/7<%9&Q<7<0$%<Q/7$'%&0%$7#0'*#-0>%9&Q<7<0$%F/79'%% 1<H>H%'C<:&[:%F/79%&0%$"<%9/.#&0%.#+%7<Y)&7<%./7<%-.<%$/%5<%C/'$g<9&$<9%$"#0%#% 67<Y)<0$%F/792%

50

slide-26
SLIDE 26
  • ]7/%#09%Z/0'D%

– ]/'$g,9&-0>%,Q/7$%5#'<9%/0%#%C#7-:)*#7%':"<.#D%

  • ]7/D%$"<%./'$%C7<:&'<%.<#')7<%
  • Z/0'D%')5L<:-@<%#09%<_C<0'&@<%

– ]/'$g,9&-0>%G&.<D%

  • ]7/D%#5*<%$/%:#$:"%:/>0&-@<%<Q/7$'%1&H<H%Z<7$#&0%<9&$'%'<<.%$/%

7<Y)&7<%./7<%:/>0&-@<%<Q/7$%$"#0%/$"<7'2%

  • Z/0'D%"&>"%@#7&#5&*&$+%

– EG,MD%%

  • ]7/D%/5L<:-@<%.<#')7<%
  • ]7/D%<#'+%$/%5<%:/.C)$<9%
  • Z/0'D%0/$%#5*<%$/%:#$:"%:/>0&-@<%<Q/7$'%%%

Quality Estimation – Turchi, FBK 51 Quality Estimation – Turchi, FBK

  • !"#$%&'%()#*&$+%,'-.#-/0%1(,23%

– 4%5&$%/6%"&'$/7+% – ()#*&$+%809&:#$/7'% – ()#*&$+%;:/7<'% – =<#70&0>%4*>/7&$".'% – ?#$#'<$% – ,@#*)#-/0%A<$7&:'% – BC<0%8'')<'%

  • ()#*&$+%,'-.#-/0%#$D%

– !/79%=<@<*% – ?/:).<0$%=<@<*%

  • E/F%:#0%F<%)'<%(,%&0%7<#*%#CC*&:#-/0'3%

52

slide-27
SLIDE 27
  • =<#70&0>%#*>/7&$".'%6/7D%

– 7<>7<''&/0D%C7<9&:$%7<#*%@#*)<'% – :*#''&[:#-/0D%C7<9&:$%&0$<><7%@#*)<'%1:#$<>/7+%*#5<*2% – 7#0R&0>D%C7<9&:$%$"<%7#0R%/6%#%'<$%/6%&$<.'%

  • P/$%#%:*<#7%<@&9<0:<%$"#$%/0<%/6%$"<.%&'%$"<%5<'$%#:7/''%9&Q<7<0$%9#$#'<$'%

#09%*#0>)#><%C#&7'%

  • A#&0%C7/5*<.'%$/%6#:<D%

– a7<#9#5*<f%./9<*% – /@<7[$%F&$"%'.#**%#./)0$%/6%9#$#%#09%*#7><%6<#$)7<%'<$%

  • %A/'$%C/C)*#7D%

– ;)CC/7$%o<:$/7%M<>7<''&/0%m;C<VWTn% – AU]%$7<<%15#'<9%/0%AU%#*>/7&$".2%m;/7VWTn% – ]#7-#*%=<#'$%;Y)#7<'%M<>7<''&/0%m;C<VTun%

% %

Quality Estimation – Turchi, FBK 53 Quality Estimation – Turchi, FBK

  • !"#$%&'%()#*&$+%,'-.#-/0%1(,23%

– 4%5&$%/6%"&'$/7+% – ()#*&$+%809&:#$/7'% – ()#*&$+%;:/7<'% – =<#70&0>%4*>/7&$".'% – ?#$#'<$% – ,@#*)#-/0%A<$7&:'% – BC<0%8'')<'%

  • ()#*&$+%,'-.#-/0%#$D%

– !/79%=<@<*% – ?/:).<0$%=<@<*%

  • E/F%:#0%F<%)'<%(,%&0%7<#*%#CC*&:#-/0'3%

54

slide-28
SLIDE 28
  • 4@#&*#5&*&$+%/6%9#$#%&'%#0%&.C/7$#0$%&'')<KK%

– ;.#**%0).5<7%/6%67<<*+%#@#&*#5*<%9#$#'<$'%% – =#5<*<9%F&$"%9&Q<7<0$%Y)#*&$+%':/7<'% – ?&Q<7<0$%*#0>)#><%C#&7'% – A/'$*+%67/.%P<F'%9/.#&0%%

  • ,4AGgTu%9#$#'<$%g%.#:"&0<%$7#0'*#-/0'%F&$"%").#0%

L)9>.<0$'%

– %WjVTTT%'<0$<0:<'%1.#:"&0<%$7#0'*#-/0k7<6<7<0:<% '<0$<0:<'2%C7/9):<9%5+%{%;AG%'+'$<.'% – Y)#*&$+%':/7<'%&0%#%hWg{i%':#*<%#'%>&@<0%C7/6<''&/0#*% $7#0'*#$/7'H% – p7g,0%#09%,0g,'%'<0$<0:<'%

Quality Estimation – Turchi, FBK 55

,4AG%Tu% ,4AG%WW% !AG%WS% =#0>)#><%]#&7'% p7g,0d%,0g,'% p7g,0d%,0g,'% ,0g,'% q%;<0$<0:<'% WjVTTT%67/.%% {%;AG'% WVTTT%,0g,'% SVUSU%p7g,0% SVSU{% ()#*&$+%;:/7<'% hWVHHHV{i% hWVIV{iV%],%G&.<V% EG<7% hWVIV%Ui% ;<0$<0:<%G+C<% $#7><$%#09% 7<6<7<0:<'% $#7><$V%C/'$g<9&$<9% #09%7<6<7<0:<'% $#7><$V%C/'$g<9&$<9% #09%7<6<7<0:<'%

Quality Estimation – Turchi, FBK 56

  • 4@#&*#5&*&$+%/6%9#$#%&'%#0%&.C/7$#0$%&'')<KK%

– ;.#**%0).5<7%/6%67<<*+%#@#&*#5*<%9#$#'<$'%% – =#5<*<9%F&$"%9&Q<7<0$%Y)#*&$+%':/7<'% – ?&Q<7<0$%*#0>)#><%C#&7'% – A/'$*+%67/.%P<F'%9/.#&0%%

slide-29
SLIDE 29
  • ,4AGgWW%9#$#'<$%g%.#:"&0<%$7#0'*#-/0'%F&$"%").#0%

L)9>.<0$'%#09%C/'$g<9&-/0'D%

– WVTTT%,0>*&'"g;C#0&'"%1,0g,'2%'<0$<0:<'% – SVUSU%p7<0:"g,0>*&'"%1p7g,02%'<0$<0:<'%% – hWg{i%Y)#*&$+%':/7<V%C/'$g<9&-0>%-.<V%#09%EG,M%':/7<%

  • !AGgWS%9#$#'<$%g%.#:"&0<%$7#0'*#-/0'%F&$"%").#0%

L)9>.<0$'%#09%C/'$g<9&-/0'D%%

– SVSU{%,0>*&'"g;C#0&'"%1,0g,'2%'<0$<0:<'% – hWgUi%Y)#*&$+%':/7<%

Quality Estimation – Turchi, FBK 57

  • !AGgWX%9#$#'<$%g%.#:"&0<%$7#0'*#-/0'%F&$"%

").#0%L)9>.<0$'%#09%C/'$g<9&-/0'%%

– ./7<%$"#0%SVSU{%,0>*&'"g;C#0&'"%1,0g,'2%'<0$<0:<'% – hWgUi%Y)#*&$+%':/7<%

– P<F%$#'R'D%

  • ]7<9&:-0>%]/'$g,9&-0>%G&.<D%./7<%$"#0%vTT%'<0$<0:<'%

F&$"%7<*#-@<%C/'$g<9&-0>%-.<%

  • !/79g*<@<*%(,D%./7<%$"#0%vTT%'<0$<0:<'%F&$"%Y)#*&$+%

#00/$#-/0%#$%F/79%*<@<*%%%%

Quality Estimation – Turchi, FBK 58

slide-30
SLIDE 30

Quality Estimation – Turchi, FBK

  • !"#$%&'%()#*&$+%,'-.#-/0%1(,23%

– 4%5&$%/6%"&'$/7+% – ()#*&$+%809&:#$/7'% – ()#*&$+%;:/7<'% – =<#70&0>%4*>/7&$".'% – ?#$#'<$% – ,@#*)#-/0%A<$7&:'% – BC<0%8'')<'%

  • ()#*&$+%,'-.#-/0%#$D%

– !/79%=<@<*% – ?/:).<0$%=<@<*%

  • E/F%:#0%F<%)'<%(,%&0%7<#*%#CC*&:#-/0'3%

59

  • 4**%$"<%&0>7<9&<0$'%6/7%5)&*9&0>%#%Y)#*&$+%<'-.#$/7D%

Quality Estimation – Turchi, FBK

– ()#*&$+%809&:#$/7'D%|)<0:+V%#9<Y)#:+V%I% – ()#*&$+%;:/7<'D%],%<Q/7$V%EG,MV%I% – =<#70&0>%4*>/7&$".'D%7<>7<''&/0V%:*#''&[:#-/0V%I% – ?#$#'<$D%!AG%STWSV%,4AG%STWWV%I%

  • E/F%9/%F<%<@#*)#$<%$"<%C<76/7.#0:<%/6%/)7%Y)#*&$+%

<'-.#$/73%

60

slide-31
SLIDE 31
  • M<>7<''&/0D%%

– A<#0%45'/*)$<%,77/7%1A4,2% – M//$%A<#0%;Y)#7<9%,77/7%1MA;,2%

  • }&@<0%#%'<$%/6%C7<9&:$<9%':/7<'%!%#09%#%'<$%/6%").#0%':/7<'%"#

Quality Estimation – Turchi, FBK

MAE = H(si) "V(si)

i=1 N

#

N RMSE = (H(si) "V(si))2

i=1 N

#

N

61

  • Z*#''&[:#-/0D%%

– ]7<:&'&/0%1]72% – M<:#**%1M<2% – 6%~':/7<%1pW2%

  • }&@<0%#%'<$%/6%C7<9&:$<9%':/7<'%!%#09%#%'<$%/6%").#0%':/7<'%"#
  • 40%<_#.C*<%6/7%5&0#7+%:*#''&[:#-/0%

Quality Estimation – Turchi, FBK

  • %

W% gW% E% W% G7)<%]/'&-@<% p#*'<%]/'&-@<% gW% p#*'<%P<>#-@<% G7)<%P<>#-@<%

Pr = tp tp + fp Re = tp tp + fn F1 = 2 " Pr" Re Pr+ Re

62

slide-32
SLIDE 32
  • ;C<#7.#0y'%M#0R%Z/<J:&<0$%
  • ?<*$#%4@<7#><%1&0$7/9):<9%#$%!AG%STWS2%

Quality Estimation – Turchi, FBK 63

E% "&% 'W% XHS% X% 'S% W% U% 'X% U% W% '{% SHw% {% 'U% {% S%

  • %

@&% 'W% U% W% 'S% W% U% 'X% {% S% '{% S% {% 'U% X% X%

System Rank Human Rank

M#0R%;&.&*#7&$+%A<$7&:%

  • M#0R&0>D%%
  • ;C<#7.#0y'%7#0R%:/<J:&<0$%12%
  • }&@<0%#%'<$%/6%C7<9&:$<9%':/7<'V%!$%#09%#%'<$%/6%").#0%':/7<'%"D%

% % % % % F"<7<%"&%7<C7<'<0$'%$"<%7#0R%/6%'<0$<0:<%'&%&0%EV%@&%7<C7<'<0$'% $"<%7#0R%/6%'<0$<0:<%'&2%&0%o%#09%% % % %#7<%$"<%#@<7#><% C/'&-/0'%&0%C7<'<0:<%/6%-<'H%%

Quality Estimation – Turchi, FBK

" = (hi # h )(vi # v )

i=1 N

$

(hi # h )2 (vi # v )2

i=1 N

$

i=1 N

$

h and v

64

slide-33
SLIDE 33
  • 80%:#'<%/6%#5'<0:<%/6%C#&7'D%
  • >&@<0%#%'<$%/6%C7<9&:$<9%':/7<'V%!$%#09%#%'<$%/6%").#0%':/7<'%"D%

% % % F"<7<%"&%7<C7<'<0$'%$"<%7#0R%/6%'<0$<0:<%'&%&0%E%#09%F"<7<%@&% 7<C7<'<0$'%$"<%7#0R%/6%'<0$<0:<%'&%&0%oH%%

  • %7#0><'%5<$F<<0%W%#09%gWH%%%%

Quality Estimation – Turchi, FBK

" =1# 6 (hi # vi)2

i=1 N

$

n(n2 #1)

65

  • ,_#.C*<D%
  • "&%7#0R%/6%'<0$<0:<%'&%&0%$"<%'<$%E%
  • @&%7#0R%/6%'<0$<0:<%'&%&0%$"<%'<$%o%%
  • 9&%9&Q<7<0:<%5<$F<<0%"&%#09%@&%

Quality Estimation – Turchi, FBK

E% "&% o% @&% 9&% 9&

S %

'W% XHS% X% U% W% gS% {% 'S% W% U% W% U% T% T% 'X% U% W% {% S% gW% W% '{% SHw% {% S% {% T% T% 'U% {% S% X% X% gW% W%

" =1# 6 di

2

$

5(52 #1) =1# 6*6 5(24) = 0.7

66

slide-34
SLIDE 34
  • ?<*$#4@<7#><D%

Quality Estimation – Turchi, FBK

E% "&% 'W% XHS% X% 'S% W% U% 'X% U% W% '{% SHw% {% 'U% {% S%

67

  • %

@&% 'W% U% W% 'S% W% U% 'X% {% S% '{% S% {% 'U% X% X%

  • ?<*$#4@<7#><D%

Quality Estimation – Turchi, FBK

E% "&% 'W% XHS% X% 'S% W% U% 'X% U% W% '{% SHw% {% 'U% {% S%

68

  • %

@&% 'W% U% W% 'S% W% U% 'X% {% S% '{% S% {% 'U% X% X%

Sc(S1) = 4 + 3 2 = 3.5 Sc(V) = 5 + 4 + 3+ 2 +1 5 = 3 DeltaAvg(1) = 3.5 " 3 = 0.5

slide-35
SLIDE 35
  • ?<*$#%4@<7#><%.<#')7<'D%

– $"<%7<*#-@<%>#&0%1*/''2%&0%C<76/7.#0:<%'<*<:-0>%$"<%$/C%<*<.<0$'%/6%$"<% $<'$%'<$%7#0R<9%#::/79&0>%$"<%C7<9&:$<9%':/7<'V%#'%$"<%C<76/7.#0:<% /5$#&0<9%67/.%$"<%$/$#*%'<$%%

  • F"<7<D%

– ;:1;2D%#@>H%@#*)<%/6%$"<%<_$7&0'&:%@#*)<'%1<H>H%],%,Q/7$2%/6%<#:"%<*<.<0$%/6%; % 1Y)#0-$#-@<%.<#')7<%/6%$"<%aY)#*&$+f%/6%$"<%'<$%;2% – ;:1;WVR2D%#@>H%@#*)<%/6%$"<%<_$7&0'&:%':/7<'%/6%$"<%<*<.<0$'%&0%$"<%[7'$%R% Y)#0-*<'%/6%;%7#0R<9%#::/79&0>%$/%$"<%C7<9&:$<9%1'+'$<.2%':/7<'% – %%0).5<7%/6%Y)#0-*<'%%%

Quality Estimation – Turchi, FBK

DeltaAvg[n] = Sc(S1,k)

k=1 n"1

#

n "1 " Sc(S)

69

  • E/F%$/%:/.C)$<%;:1;WVR23%

– 7#0R%$"<%<*<.<0$'%/6%;%#::/79&0>%$"<%C7<9&:$<9%1'+'$<.2%':/7<'% – 6/7%<*<.<0$%&0%$"<%R%Y)#0-*<'V%').%$"<%<_$7&0'&:%':/7<'%#09%#@<7#><%&$%

  • 6/7%%z%W%%

– ?<*$#4@>mWn%z%;:1;W2%~;:1;2%

  • 6/7%%%zS%

– ?<*$#4@>mSn%z%1;:1;W2%~;:1;WVS22kS%~%;:1;2%

  • 49@#0$#><'D%

– "&>"%:/77<*#-/0%F&$"%$"<%;C<#7.#0y'%:/<J:&<0$% – .<#')7<'%$"<%Y)#*&$+%/6%#%"+C/$"<'&'%7#0R&0>%67/.%#0%<_$7&0'&:%C<7'C<:-@< % 1#'%/Q<7<9%5+%6)0:-/0%o2%

Quality Estimation – Turchi, FBK

`%6/7%./7<%9<$#&*%'<<%mZ#*VWSn%

70

slide-36
SLIDE 36
  • !"&:"%&'%$"<%.<#0&0>%/6%THU3%

– 9&Q<7<0:<%&0%Y)#*&$+%5<$F<<0%$"<%$/Cg7#0R<9%Y)#0-*<'%#09%$"<% /@<7#**%Y)#*&$+%&'%THU%

  • 8'%THU%#%>//9%':/7<3%

– PT%#%C7&/7&%)CC<75/)09%

  • G"<%)CC<75/)09%9<C<09'%/0%$"<%$<'$%'<$%

– 8$%7<Y)&7<'%#0%B7#:*<%;:/7<%$/%:/.C#7<%F&$"%

  • 5#'<9%/0%EG,M%/7%E).#0%L)9>.<0$'%

– 80%/)7%:#'<V%B7#:*<%?<*$#4@>1E).#0%l)9>.<0$'V%[7'$%Y)#0-*<2%z% 1UÄ{2kS%gX%z%WHU%%

Quality Estimation – Turchi, FBK 71 Quality Estimation – Turchi, FBK

  • !"#$%&'%()#*&$+%,'-.#-/0%1(,23%

– 4%5&$%/6%"&'$/7+% – ()#*&$+%809&:#$/7'% – ()#*&$+%;:/7<'% – =<#70&0>%4*>/7&$".'% – ?#$#'<$% – ,@#*)#-/0%A<$7&:'% – BC<0%8'')<'%

  • ()#*&$+%,'-.#-/0%#$D%

– !/79%=<@<*% – ?/:).<0$%=<@<*%

  • E/F%:#0%F<%)'<%(,%&0%7<#*%#CC*&:#-/0'3%

72

slide-37
SLIDE 37
  • =#:R%/6%#0%/5L<:-@<%Y)#*&$+%':/7<%#5*<%$/%:#$:"%

:/>0&-@<%<Q/7$'%

– 4%0<F%':/7<%$"#$%:/0$#&0'%$"<%.#&0%6<#$)7<'%/6%EG,M%#09% :/77<*#$<'%F<**%F&$"%],%-.<%

  • =#:R%/6%#%$<:"0&Y)<%#5*<%$/%$"7<'"/*9%$"<%Y)#*&$+%

':/7<%15#9%@'H%>//9%$7#0'*#-/0'2%

– 8'%EG,M%z%THXkTHUkTHw%#%5#9%/7%>//9%$7#0'*#-/03% – O'<6)*%&0%$"<%Z4G%$//*%':<0#7&/V%F"<7<%&$%&'%0<:<''#7+%$/% 9&':#79%5#9%$7#0'*#-/0'%

Quality Estimation – Turchi, FBK 73

  • A/7<%$"#0%WVTTT%Y)#*&$+%&09&:#$/7'%"#@<%5<<0%9<@<*/C<9%

&0%$"<%*#'$%+<#7'H%

– ?/%F<%0<<9%#**%/6%$"<.%&0%#%7<#*%#CC*&:#-/03% – !"&:"%#7<%$"<%./'$%7<*&#5*<%&0%<#:"%>7/)C3% – !"&:"%&'%$"<%5<'$%:/.5&0#-/03%

  • ;)5L<:-@&$+%&0%$"<%C/'$g<9&$/7%F/7R%#09%&0%$"<%$#'R%

– 4%'&0>*<%Y)#*&$+%<'-.#$/7%6/7%@<7+%9&Q<7<0$%C/'$g<9&$/7% 5<"#@&/7'%#09%$#'R'% – ]<7'/0#*&b#-/0%

  • I%%

Quality Estimation – Turchi, FBK 74

slide-38
SLIDE 38

Quality Estimation – Turchi, FBK

  • !"#$%&'%()#*&$+%,'-.#-/0%1(,23%

– 4%5&$%/6%"&'$/7+% – ()#*&$+%809&:#$/7'% – ()#*&$+%;:/7<'% – =<#70&0>%4*>/7&$".'% – ?#$#'<$% – ,@#*)#-/0%A<$7&:'% – BC<0%8'')<'%

  • ()#*&$+%,'-.#-/0%#$D%

– !/79%=<@<*% – ?/:).<0$%=<@<*%

  • E/F%:#0%F<%)'<%(,%&0%7<#*%#CC*&:#-/0'3%

75

  • ]7<9&:$%$"<%Y)#*&$+%':/7<%/6%<#:"%F/79%&0%$"<%$#7><$%'<0$<0:<%

– 5&0#7+D%>//9k5#9% – .)*-g:*#''%

  • '<Y)<0-#*%*<#70&0>%$#'RD%%

– C7<9&:-/0%/6%F/79%L%9<C<09'%/0%$"<%C7<9&:-/0%/6%F/79%LgW%1./7<% :/.C*<_%./9<*'%:#0%5<%)'<92%

  • '&0>*<%F/79%:*#''&[:#-/0D%%

– <#:"%F/79%C7<9&:-/0%&'%&09<C<09<0$%67/.%$"<%/$"<7%

  • !/79%C7<9&:-/0'%)'<6)*D%

– #'%6<#$)7<'%6/7%$"<%'<0$<0:<%*<@<*%Y)#*&$+%<'-.#-/0% – $/%C7<9&:$%<77/7%$+C<'V%<H>H%80'<7-/0V%?<*<-/0V%I% – $/%"<*C%$7#0'*#$/7'%$/%5<s<7%&9<0-6+%F7/0>%C#7$%/6%$"<%'<0$<0:<%

<H>%$"<%:#$%F<7<%/0%$"<%<#>*<%

Quality Estimation – Turchi, FBK 76

slide-39
SLIDE 39
  • }//90<''%mN#:VWWnD%

– [09%$"<%5<'$%'<Y)<0:<%/6%F/79%*#5<*'%

  • 4%'<Y)<0-#*%*#5<*&0>%$#'R%%
  • O'&0>%9&Q<7<0$%*#5<*'D%

– 5&0#7+D%N#9k}//9%%/7% – .)*-g:*#''D%80'<7-/0k;)5'-$)-/0k;"&\k}//9%H%

  • =#7><%#./)0$'%/6%'C#7'<%5&0#7+%6<#$)7<'%6/7%F/79%

C7<9&:-/0D%

– '/)7:<%&06/7.#-/0k]B;k:/0$<_$%1<H>H%0<_$%$F/%F/79'2k #*&>0.<0$%:/0$<_$k:/.5&0#-/0%/6%#*&>0.<0$%#09%]B;k 9<C<09<0:+%'$7):$)7<%1#$%F/79%#09%C"7#'<%*<@<*'2%

Quality Estimation – Turchi, FBK 77

  • G7#&0&0>%)'&0>%$"<%A#7>&0%806)'<9%M<*#_<9%4*>/7&$".%1A8M42%

– <J:&<0$%.<:"#0&'.%$/%$7#&0%#09%/C-.&b<%*<#70&0>%'+'$<.'%F&$"% */$'%/6%'C#7'<%6<#$)7<'% – /0*&0<%

  • BC-.#*%'<Y)<0:<%/5$#&0<9%5+%$"<%o&$<75&%#*>/7&$".%
  • G7#&0&0>%9#$#D%wUVTTT%'<0$<0:<%C#&7'%
  • ;/)7:<%'&9<%&06/7.#-/0V%#*&>0.<0$%:/0$<_$V%#09%

9<C<09<0:+%'$7):$)7<%6<#$)7<'%#**/F%&.C7/@<.<0$'%&0% 5&0#7+%#09%.)*-g:*#''%C7<9&:-/0'%%

Quality Estimation – Turchi, FBK 78

slide-40
SLIDE 40

Quality Estimation – Turchi, FBK

  • !"#$%&'%()#*&$+%,'-.#-/0%1(,23%

– 4%5&$%/6%"&'$/7+% – ()#*&$+%809&:#$/7'% – ()#*&$+%;:/7<'% – =<#70&0>%4*>/7&$".'% – ?#$#'<$% – ,@#*)#-/0%A<$7&:'% – BC<0%8'')<'%

  • ()#*&$+%,'-.#-/0%#$D%

– !/79%=<@<*% – ?/:).<0$%=<@<*%

  • E/F%:#0%F<%)'<%(,%&0%7<#*%#CC*&:#-/0'3%

79

  • ,_$<09%$"<%'<0$<0:<%*<@<*%(,%$/%9/:).<0$%*<@<*%
  • ]7<9&:$<9%Y)#*&$+%':/7<%6/7%<#:"%'&0>*<%F"/*<%9/:).<0$

%

  • M#0R%$7#0'*#$<9%9/:).<0$'%67/.%>//9%$/%5#9%)'&0>%

9/:).<0$%*<@<*%(,%C7<9&:-/0'%

  • p<F%F/7R%"#@<%&0@<'->#$<9%(,%#$%9/:).<0$%*<@<*D%

– 0<F%$#'R% – ./'$%/6%$"<%(,%7<'/)7:<'%#$%'<0$<0:<%*<@<*% – )'<7g/7&<0$<9%17<#*%7<Y)<'$%5+%*#0>)#><%'<7@&:<%C7/@&9<7'2%

Quality Estimation – Turchi, FBK 80

slide-41
SLIDE 41
  • G7)'$M#0R%m;/7VWTnD%

– $7#0'*#$<9%9/:).<0$'%7#0R<9%#::/79&0>%$"<%Y)#*&$+%':/7<%1N=,O2% C7<9&:-/0'% – C7<9&:-/0%#$%9/:).<0$%*<@<*%

  • N#'<9%/0%

– $"<%AU]%$7<<%7<>7<''&/0%#*>/7&$".% – *#7><%'<$%/6%'/)7:<%#09%$#7><$%9/:).<0$%*<@<*%6<#$)7<'D%

  • $<_$k*#0>)#><%./9<*k<_#.C*<%1<H>H%'&.&*#7&$+%F&$"%C7<g*#5<*<9%

9/:).<0$'2kC'<)9/%7<6<7<0:<k$7#&0&0>%9#$#%~5#'<9%

  • (,%C<76/7.#0:<%%

– <77/7%'.#**<7%$"#0%$"<%5#'<*&0<%5)$%'-**%*#7><%1ÅvHU%N=,O%':/7<% C/&0$'2%

Quality Estimation – Turchi, FBK 81

  • M<*<@#0:<%M#0R&0>%6/7%G7#0'*#$<9%G<_$'%mG)7VWSnD%

– $"<%./'$%7<*<@#0$%9/:).<0$%'"/)*9%"#@<%$"<%./'$%&.C/7$#0$% '<0$<0:<'%:/77<:$*+%$7#0'*#$<9% – 9/:).<0$%C7<9&:-/0%z%#@>H%:/0$7&5)-/0%/6%#**%$"<%'<0$<0:<'%

  • (,%':/7<%5#'<9%/0%$"<%:/.C*<_&$+%#09%|)<0:+%6<#$)7<'%
  • M<*<@#0:<%7#0R<7'D%

– )0')C<7@&'<9%:/g/::)77<0:<g5#'<9%1'<0$<0:<'%7#0R<9%#::/79&0>%:/g /::)77<0:<%/6%$<7.'%&0%$"<%9/:).<0$2% – ')C<7@&'<9%C7/[*<%5#'<9%1'<0$<0:<'%7#0R<9%#::/79&0>%$"<%:/'&0<% '&.&*#7&$+%F&$"%#%C7/[*<%&0%$"<%9/.#&02%

  • E&>"%C<76/7.#0:<%@#7&#5&*&$+%#:7/''%9&Q<7<0$%9/.#&0'%%

– @<7+%>//9%&0%'/.<%9/.#&0'V%'*&>"$*+%5<s<7%$"#0%$"<%5#'<*&0<%&0%/$"<7'%

Quality Estimation – Turchi, FBK 82

SentenceScore =QE prediction " relevanceranking

slide-42
SLIDE 42

Quality Estimation – Turchi, FBK

  • !"#$%&'%()#*&$+%,'-.#-/0%1(,23%

– 4%5&$%/6%"&'$/7+% – ()#*&$+%809&:#$/7'% – ()#*&$+%;:/7<'% – =<#70&0>%4*>/7&$".'% – ?#$#'<$% – ,@#*)#-/0%A<$7&:'% – BC<0%8'')<'%

  • ()#*&$+%,'-.#-/0%#$D%

– !/79%=<@<*% – ?/:).<0$%=<@<*%

  • E/F%:#0%F<%)'<%(,%&0%7<#*%#CC*&:#-/0'3%

83 Quality Estimation – Turchi, FBK 84

slide-43
SLIDE 43
  • }//>*<%G7#0'*#$<%

Quality Estimation – Turchi, FBK 85

  • p#:<5//R%
  • P<F'%G7#0'*#-/0%M<#9<7%

Quality Estimation – Turchi, FBK 86

slide-44
SLIDE 44

Marco Turchi turchi@fbk.eu

Statistical Machine Translation - ICT Doctoral School – 19 March 2013

Quality Estimation – Turchi, FBK 87

  • mlEOVTXn%N*#$bV%lHV%,H%p&$b><7#*9V%}H%p/'$<7V%;H%}#097#5)7V%ZH%}/)s<V%4H%

x)*<'b#V%4H%;#0:"&'V%#09%PHH%O<J0>V%!"#$%&#'&()*+,-+"#(."/(0-'12#&( 3/-#*4-+"#V%V%N#*-./7<V%p&0#*%7<C/7$%lEO%k%Z=;]%STTX%;)..<7%!/7R'"/CV% STT{H%

  • m;/7VWTnM#9)%;/7&:)$%#09%459<''#.#9%,:"&"#5&H%STWTH%3/5*6/-#78(9#%5'2#:(

6/5*6(2#(-56",-+'(6/-#*4-+"#*(;2-(/-#72#:H%80%]7/:<<9&0>'%/6%4Z=H%

  • mZ#*VWSn%Z"7&'%Z#**&'/0gN)7:"V%]"&*&CC%x/<"0V%Z"7&'$/6%A/0bV%A#s%]/'$V%

M#9)%;/7&:)$V%#09%=):&#%;C<:&#H%STWSH%<2#%2#:*(".(61&(=>?=(@"/7*1"A("#( *6-+*+'-4(,-'12#&(6/-#*4-+"#H%80%]7/:<<9&0>'%/6%$"<%;<@<0$"%!/7R'"/C% /0%;$#-'-:#*%A#:"&0<%G7#0'*#-/0%V%A/0$7<#*V%Z#0#9#V%l)0<H%4''/:&#-/0% 6/7%Z/.C)$#-/0#*%=&0>)&'-:'H%

  • m;C<VWTn%;C<:&#V%=):&#V%?"F#L%M#LV%#09%A#7:/%G)7:"&H%0-'12#&(6/-#*4-+"#(

&;-45-+"#(;&/*5*(B5-426C(&*+,-+"#H%A#:"&0<%$7#0'*#-/0%S{HW%1STWT2D% XugUTH%

Quality Estimation – Turchi, FBK 88

slide-45
SLIDE 45
  • m;C<VTun%;C<:&#V%=HV%G)7:"&V%AHVZ#0:<99#V%PHV%?+.<$.#0V%AH%V%Ç%Z7&'-#0&0&V%

PH%1STTu2H%)*+,-+#:(61&(*&#6&#'&D4&;&4(B5-426C(".(,-'12#&(6/-#*4-+"#( *C*6&,*H%80%&'(#)*+'#,%%-./#01%2(3(%4(#12#+'(#5-316(.%#,77148.91%#213# :.4'8%(#&3.%7/.91%#;5,:&<=>>?@#;66A#=B<*C@A#

  • mByN7VWTn%ByN7&<0V%;H%9#6/"%5'+"#(6"(E"*6D)%2+#:8(@1"F(@1-6F(1"@(-#%(

@1&/&(6"(#&G6H(G)$/7&#*%#$%4AG4V%STWT%

  • mN#:VWWn%N#:"V%P>)+<0V%p<&%E)#0>V%#09%É#'<7%4*gB0#&b#0H%I""%#&**8(J(

,&61"%(."/(,&-*5/2#:(,-'12#&(6/-#*4-+"#('"#$%&#'&H%]7/:<<9&0>'%/6% $"<%{u$"%400)#*%A<<-0>%/6%$"<%4''/:&#-/0%6/7%Z/.C)$#-/0#*%=&0>)&'-:'D% E).#0%=#0>)#><%G<:"0/*/>&<'H%o/*H%WH%STWWH%

  • mG)7VWSn%G)7:"&V%A#7:/V%l/'<6%;$<&05<7><7V%#09%=):&#%;C<:&#H%K&4&;-#'&(

K-#72#:(."/(3/-#*4-6&%(3&G6*H%80%]7/:<<9&0>'%/6%,4AG%STWSH%%

Quality Estimation – Turchi, FBK 89