SLIDE 1
Combining Crowd and AI to scale professional-quality translation
João Graça CTO
80%
English
Combining Crowd and AI to scale professional-quality translation - - PowerPoint PPT Presentation
Combining Crowd and AI to scale professional-quality translation - - PowerPoint PPT Presentation
Combining Crowd and AI to scale professional-quality translation Joo Graa CTO The Internet, 1997 80% English The Internet, 2017 30% English 20% Chinese 8% 6% Spanish 5% Japanese 4% Portuguese 3% German
SLIDE 2
SLIDE 3
5%
Portuguese
8%
Spanish
20%
Chinese
4%
German
6%
Japanese
3%
Arabic
30%
English
The Internet, 2017
SLIDE 4
“All translation firms together are able to translate far less than 1% of relevant content produced everyday”
CSA – MT Is Unavoidable to Keep Up with Content Volumes
SLIDE 5
Is Machine Translation already here?
*
Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions Marcin Junczys-Dowmunt, Tomasz Dwojak, Hieu Hoang
Everyone agrees that NMT is here to stay and much better than SMT
SLIDE 6
Bleu Moses (generic) Neural MT (generic) Moses (adapted) Neural MT (adapted) 49,9 43,5 35,7 29,6
Unbabel Experiments with Customer Service Tickets
SLIDE 7
Is NMT really enough?
* A Neural Network for Machine Translation, at Production Scale. Google Blog
SLIDE 8
0% 20% 40% 60% 80% 100% 6 12 18 24 30 Good Not sure Bad
MQM
Job
Quality per Job
SLIDE 9
Will AI solve translation?
MACHINE ONLY TIME JOBS MQM 95 QUALITY
SLIDE 10
Will AI solve translation?
HUMAN EFFORT TIME JOBS MQM 95
SLIDE 11
0% 20% 40% 60% 80% 100% 6 12 18 24 30 Good Not sure Bad
MQM
Job
Quality per Job
SLIDE 12
Unbabel Pipeline
SLIDE 13
Job
Unbabel Pipeline
SLIDE 14
Job Machine Translation
Unbabel Pipeline
SLIDE 15
Job Machine Translation
Unbabel Pipeline
SLIDE 16
Job Machine Translation
Unbabel Pipeline
Q.E.
Quality Estimation
SLIDE 17
Job Machine Translation Customer High Q.E.
Unbabel Pipeline
Q.E.
Quality Estimation
SLIDE 18
Job Machine Translation Customer High Q.E.
Unbabel Pipeline
Q.E.
Quality Estimation Low Q.E. Re-Eval Community Translators
SLIDE 19
Job Machine Translation Customer
Q.E.
Quality Estimation Community Translators Customer
Q.E.
Quality Estimation
Data Generation Engine
SLIDE 20
Initial text Submitted text
After
Initial text Submitted text
Before
NO DATA POINTS
All changes in between:
Mouse clicks Key presses Timestamps
Data Generation Engine
SLIDE 21
Raw data Processed information
At 18:03:30: In nugget 3 mouseClick Cursor at 16 Selected: 0 At 18:03:31: In nugget 3 Pressed Backspace Cursor at 16 Selected: 0 At 18:03:31: In nugget 3 Pressed Backspace Cursor at 15 Selected: 0 At 18:03:31: In nugget 3 Pressed Backspace At 18:03:35: In nugget 3 Pressed Shift Cursor at 25 Selected: 0 At 18:03:35: In nugget 3 Pressed s Cursor at 25 Selected: 0 At 18:03:35: In nugget 3 Pressed i Cursor at 26 Selected: 0 At 18:03:35: In nugget 3 Pressed e At 18:03:30: In nugget 3 mouseClick Cursor at 16 Selected: 0 At 18:03:31: In nugget 3 Pressed Backspace Cursor at 16 Selected: 0 At 18:03:31: In nugget 3 Pressed Backspace Cursor at 15 Selected: 0 At 18:03:31: In nugget 3 Pressed Backspace
Initial text
“Espero que esto es útil”
- Deleted word “es”
- Inserted word “sea”
Submitted text
“Espero que esto sea útil”
Keystroke Analysis
SLIDE 22
Job Machine Translation Customer High Q.E.
Q.E.
Quality Estimation Low Q.E. Re-Eval Crowd
MACHINE QE COMMUNITY
Unbabel Pipeline
SLIDE 23
Translation Memory Job Result MT Router Customer MT Customer APE
Machine Translation Pipeline
SLIDE 24
Phrase-based MT Neural MT
Machine Translation Models
SLIDE 25
Customer Support Tickets
30 37,5 45 52,5 60
G
- g
l e N M T N M T 1 K 5 K 1 K 2 k 2 5 K 51,7 48,8 47,2 46,9 43,2 34,1 43,1
Bleu
Customer Adaptation
SLIDE 26
Word-Level QE Which words are translated correctly/incorrectly? Sentence-Level QE How good is the entire translation?
Quality Estimation
SLIDE 27
Word-level QE example
Hey lá , eu sou pesaroso sobre aquele !
OK OK OK OK BA D BA D BA D BA D BA D
Quality Estimation
SLIDE 28
Unbabel Ticket Bad translation Good translation
source MT final
QE Training
SLIDE 29
QE Word Level
SLIDE 30
Ugent System LinearQE NeuralQE StackedQE APE-QE FullStackedQE
15 30 45 60F1_MULT
QE Word Level
SLIDE 31
Ugent System LinearQE NeuralQE StackedQE APE-QE FullStackedQE
15 30 45 60F1_MULT
41,1
QE Word Level
SLIDE 32
Ugent System LinearQE NeuralQE StackedQE APE-QE FullStackedQE
15 30 45 60F1_MULT
45,2 41,1
QE Word Level
SLIDE 33
Ugent System LinearQE NeuralQE StackedQE APE-QE FullStackedQE
15 30 45 60F1_MULT
47,3 45,2 41,1
QE Word Level
SLIDE 34
Ugent System LinearQE NeuralQE StackedQE APE-QE FullStackedQE
15 30 45 60F1_MULT
50,3 47,3 45,2 41,1
WMT 16 WL QE Winner
QE Word Level
SLIDE 35
Ugent System LinearQE NeuralQE StackedQE APE-QE FullStackedQE
15 30 45 60F1_MULT
55,7 50,3 47,3 45,2 41,1
WMT 16 WL QE Winner
QE Word Level
SLIDE 36
Ugent System LinearQE NeuralQE StackedQE APE-QE FullStackedQE
15 30 45 60F1_MULT
57,5 55,7 50,3 47,3 45,2 41,1
Andre F. T. Martins, Marcin Junczys-Dowmunt, Fabio Kepler, Ramon Astudillo, Chris Hokamp, Roman Grundkiewicz. “Pushing the Limits of Translation Quality Estimation.” TACL 2017 (To Appear)
WMT 16 WL QE Winner
QE Word Level
SLIDE 37
QE Sentence Level
SLIDE 38
YANDEX StackedQE APE-QE FullStackedQE
Pearson Correlation
17,5 35 52,5 70QE Sentence Level
SLIDE 39
YANDEX StackedQE APE-QE FullStackedQE
Pearson Correlation
17,5 35 52,5 7052,5
QE Sentence Level
SLIDE 40
YANDEX StackedQE APE-QE FullStackedQE
Pearson Correlation
17,5 35 52,5 7054,9 52,5
QE Sentence Level
SLIDE 41
YANDEX StackedQE APE-QE FullStackedQE
Pearson Correlation
17,5 35 52,5 7061,3 54,9 52,5
QE Sentence Level
SLIDE 42
YANDEX StackedQE APE-QE FullStackedQE
Pearson Correlation
17,5 35 52,5 7065,6 61,3 54,9 52,5
QE Sentence Level
Andre F. T. Martins, Marcin Junczys-Dowmunt, Fabio Kepler, Ramon Astudillo, Chris Hokamp, Roman Grundkiewicz. “Pushing the Limits of Translation Quality Estimation.” TACL 2017 (To Appear)
SLIDE 43
QE in the Pipeline
SLIDE 44
Job Machine Translation Customer High Q.E.
Q.E.
Quality Estimation
QE in the Pipeline
SLIDE 45
Job Machine Translation Customer High Q.E.
Q.E.
Quality Estimation
Document-Level QE how good is the entire document?
QE in the Pipeline
SLIDE 46
Job Machine Translation Customer High Q.E.
Q.E.
Quality Estimation Low Q.E. Re-Eval Community Translators
Document-Level QE how good is the entire document?
QE in the Pipeline
SLIDE 47
Job Machine Translation Customer High Q.E.
Q.E.
Quality Estimation Low Q.E. Re-Eval Community Translators
Document-Level QE how good is the entire document? Human QE Can we evaluate post-edit output?
Interesting numbers coming soon
QE in the Pipeline
SLIDE 48
Goals Quality Speed Cost
Professional Translation
SLIDE 49
Goals Quality Speed Cost Pillars
Professional Translation
SLIDE 50
Goals Quality Speed Cost Pillars
Professional Translation
SLIDE 51
Goals Quality Speed Cost Pillars
- Editors Pool
Professional Translation
SLIDE 52
Goals Quality Speed Cost Pillars
- Editors Pool
- Initial Text (MT)
Professional Translation
SLIDE 53
Goals Quality Speed Cost Pillars
- Editors Pool
- Initial Text (MT)
- Editor Assignment
Professional Translation
SLIDE 54
Goals Quality Speed Cost Pillars
- Editors Pool
- Initial Text (MT)
- Editor Assignment
- Interfaces
Professional Translation
SLIDE 55
Goals Quality Speed Cost Pillars
- Editors Pool
- Initial Text (MT)
- Editor Assignment
- Interfaces
- Quality Evaluation
Professional Translation
SLIDE 56
Unbabel Community
SLIDE 57
50 000 Users
Unbabel Community
SLIDE 58
Quality Estimation
Distributed Pipeline
SLIDE 59
Expert
Specialisation layers will grow with time
Testing Phase
Editors are tested when they sign up
Training Content
Editors get ratings for the tasks
Paid Content
The best editors have access to paid content
Editors Pool
SLIDE 60
Expert
Specialisation layers will grow with time
Testing Phase
Editors are tested when they sign up
Training Content
Editors get ratings for the tasks
Paid Content
The best editors have access to paid content
Evaluators
Editors Pool
SLIDE 61
Expert
Specialisation layers will grow with time
Testing Phase
Editors are tested when they sign up
Training Content
Editors get ratings for the tasks
Paid Content
The best editors have access to paid content
Evaluators Annotators
Editors Pool
SLIDE 62
How Editors are Evaluated
SLIDE 63
Editors Profiling
SLIDE 64
Editors Profiling
SLIDE 65
Editor Assignment
SLIDE 66
Tasks/time
6 m 2 m 10 m 12 m 18 m 45 m
Editor Assignment
SLIDE 67
Editors Tasks/time
6 m 2 m 10 m 12 m 18 m 45 m
Editor Assignment
SLIDE 68
Pull
Editors Tasks/time
6 m 2 m 10 m 12 m 18 m 45 m
Editor Assignment
SLIDE 69
Pull
30 m 6 H 2 D 6 D 20 m 40 m
SLA Editors Tasks/time
6 m 2 m 10 m 12 m 18 m 45 m
Editor Assignment
SLIDE 70
Pull
30 m 6 H 2 D 6 D 20 m 40 m
SLA
1100 1000 1000 1000 1100 1100
Priority Editors Tasks/time
6 m 2 m 10 m 12 m 18 m 45 m
Editor Assignment
SLIDE 71
Pull
30 m 6 H 2 D 6 D 20 m 40 m
SLA
1100 1000 1000 1000 1100 1100
Priority Editors Queue
G G G G R R
Tasks/time
6 m 2 m 10 m 12 m 18 m 45 m
Editor Assignment
SLIDE 72
Pull
30 m 6 H 2 D 6 D 20 m 40 m
SLA
1100 1000 1000 1000 1100 1100
Priority Editors Rating
4.2 3.8 4.3 4.8
Queue
G G G G R R
Tasks/time
6 m 2 m 10 m 12 m 18 m 45 m
Editor Assignment
SLIDE 73
Pull
30 m 6 H 2 D 6 D 20 m 40 m
SLA
1100 1000 1000 1000 1100 1100
Priority Editors Rating
4.2 3.8 4.3 4.8
Native Queue
G G G G R R
Tasks/time
6 m 2 m 10 m 12 m 18 m 45 m
Editor Assignment
SLIDE 74
Pull
30 m 6 H 2 D 6 D 20 m 40 m
SLA
1100 1000 1000 1000 1100 1100
Priority Editors Rating
4.2 3.8 4.3 4.8
Native Queue
G G G G R R
Tasks/time
6 m 2 m 10 m 12 m 18 m 45 m
Topics
Editor Assignment
SLIDE 75
Pull
30 m 6 H 2 D 6 D 20 m 40 m
SLA
1100 1000 1000 1000 1100 1100
Priority Editors Rating
4.2 3.8 4.3 4.8
Native Topics Queue
G G G G R R
Tasks/time
6 m 2 m 10 m 12 m 18 m 45 m
Topics
Editor Assignment
SLIDE 76
Editor Assignment
SLIDE 77
Regular distribution
3.8
- ld rating
Editor Assignment
SLIDE 78
Regular distribution
3.8
- ld rating
Smart distribution
4.6
Improved rating
Editor Assignment
SLIDE 79
Post-Editing Interfaces
SLIDE 80
Time Spent on Job
TIME
DELIVERY
MT
SLIDE 81
Time Spent on Job
WAITING
Translator 1 TIME
DELIVERY
MT
SLIDE 82
Time Spent on Job
WAITING
Translator 1 TIME
DELIVERY
MT Translator 2
WAITING
SLIDE 83
Time Spent on Job: Mobile
WAITING
Translator 1 TIME
DELIVERY
MT Translator 2
WAITING
20%
less
SLIDE 84
Time Spent on Job
SLIDE 85
Pos-Editing Interfaces
SLIDE 86
Smartcheck
SLIDE 87
Spelling Tone Formality Consistency
External NLP Services
Spell Check Syntax Parser Word Aligner
Client Rule
Smartcheck
SLIDE 88
Spelling Tone Formality Consistency
External NLP Services
Spell Check Syntax Parser Word Aligner
Client Rule
Smartcheck
SLIDE 89
Spelling Tone Formality Consistency
External NLP Services
Spell Check Syntax Parser Word Aligner
Client Rule
Smartcheck
Annotation Tool
Eval
Annotated
SLIDE 90
Spelling Tone Formality Consistency
External NLP Services
Spell Check Syntax Parser Word Aligner
Client Rule
Smartcheck
Annotation Tool
Eval
Annotated
Learn
SLIDE 91
M A C H I N E + H U M A N
CORE API CHAT API CYRANO API
Customer Service Conversational
MESSAGING API
Language OS
Language Engine
Bots
SLIDE 92
Tickets can come in many languages.
In Customer Service
SLIDE 93
Unbabel for Zendesk
Unbabel’s Domain Adapted Machine Translation Distributed Human Translation English-speaking agent Zendesk app connected with Unbabel API
Unbabel can offer the same Customer Satisfaction as native agents
SLIDE 94
SPEED QUALITY
94
20
minutes
Customer Replies: Speed & Quality
SLIDE 95
Chat
SLIDE 96
Editors train the MT engine Community of 50K+ translators Skips Humans
Chat Translation Flow
SLIDE 97
Chat Messages: Speed & Quality
SPEED QUALITY
90
2
Minutes
SLIDE 98
What is the future?
HUMAN EFFORT TIME JOBS MQM 95
0%?
SLIDE 99
Thank You
joao@unbabel.com