Background Video websites (e.g. YouTube) have proliferated - - PDF document

10/6/2011 1

The Importance of Visual Context Clues in Multimedia Translation

Christopher G. Harris The University of Iowa, USA Tao Xu Tongji University, China

Background

• Video websites (e.g. YouTube) have proliferated

– estimated 1 billion views daily in 2010

• Quick translations essential to reaching wider

global audience

• Professional translators: Expensive and slow
• Machine Translation Tools: Cheap… but

accurate?

• What about crowdsourcing?

CLEF 2011 Amsterdam

2

10/6/2011 2

Research objectives

• Is it sufficient to work from a written

transcript… or are the visual context clues found in video beneficial for translation?

• Since crowdsourcing involves humans (who

can take advantage of video context), how effective is crowdsourcing relative to the MT tools available today?

• Does our success depend on the genre of

multimedia we choose to translate?

CLEF 2011 Amsterdam

3

Videos Examined

Score CS and MT results against a gold standard (PT) using translations from Mandarin Chinese to:

• Spanish
• Russian
• English

Meteor can match 3 different ways:

• Exact match
• Stemmed match
• Synonym match (powerful and flexible)

Meteor evaluation tool

CLEF 2011 Amsterdam

4

10/6/2011 3

Videos Examined

= ∙ ∙ + 1 − ∙ = ∙ ( / )

Meteor parameters

CLEF 2011 Amsterdam

English Russian Spanish  0.95 0.85 0.90  0.50 0.60 0.50  0.45 0.70 0.55

tradeoff between precision & recall maximum penalty functional relation between fragmentation and penalty

5

Experimental setup (1)

CLEF 2011 Amsterdam Video MT MT MT Machine Translation The Crowd Professional Translator Transcript

6

10/6/2011 4

Experimental setup (2)

CLEF 2011 Amsterdam Meteor MT MT MT Machine Translation The Crowd Professional Translator Score

7

The bigger picture

CLEF 2011 Amsterdam

We want to:

• Compare 3 genres (AN, TS, MV)
• Use 3 videos from each genre
• Compare 3 languages (EN, ES, RU)
• Test with and without video usage
• 3 x 3 x 3 x 2 x 2 = 108 runs

For each run:

• 5 MT tools
• Minimum of 2 CS translations

( = 3. 8 translations/transcript)

8

Video MT The Crowd PT Transcript

Meteor Score

10/6/2011 5

Videos Examined

MT tools used

• Google translate
• Babelfish
• Bing
• Worldlingo
• Lingoes*

Tools and platforms used

CLEF 2011 Amsterdam

9

CS platforms used

• oDesk
• eLance
• Taskcn*
• Zhubajie*
• epWeike*

* = Chinese-based (No Mechanical Turk)

Videos Examined

3 animated clips

• Plenty of exaggerated

expressions

• Use of imagery

Animation video clips examined

CLEF 2011 Amsterdam

10

10/6/2011 6

Videos Examined

3 music videos

• Figurative poetic language
• Uses lots of imagery

Music video clips examined

CLEF 2011 Amsterdam

11

Videos Examined

3 talk show clips

• Fast-paced dialog
• Lots of sarcasm and

idiomatic expressions used

Talk show video clips examined

CLEF 2011 Amsterdam

12

10/6/2011 7

Meteor scoring

CLEF 2011 Amsterdam

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00

13

Videos Examined Features evaluated

CLEF 2011 Amsterdam

Multimedia Genre Translation Type CS MT PT

MV AN

TS

14

• Four features studied
• Use a cube to represent

three of them

• Language
• Genre
• Type of translation

method

• MM vs. transcript

evaluated separately

10/6/2011 8

Videos Examined Representing our results

CLEF 2011 Amsterdam

Multimedia Genre Translation Type

AN

MV

TS CS MT

15

Videos Examined Heat-map showing Meteor score

CLEF 2011 Amsterdam

16

10/6/2011 9

Videos Examined Where did visual context help most?

CLEF 2011 Amsterdam

17

Videos Examined Inter-annotator agreement

CLEF 2011 Amsterdam

MM WT TS 0.69 0.61 AN 0.71 0.67 MV 0.65 0.57

18

• Inter-annotator agreement (Cohen’s Kappa) between

crowdsourced and professional translations grouped by genre.

• The MM considers visual context clues whereas WT
• nly considers the written transcripts.

Genre Media source used

10/6/2011 10

Videos Examined Additional validation

CLEF 2011 Amsterdam

19

Validated translations at a high level, we had human translators provide simple preference judgments on each feature:

• Crowdsourced translations generated from written

transcripts compared with crowdsourced translations generated from multimedia

• Machine translations compared with crowdsourced

translations

• Professional translations compared with the

crowdsourced translations

Videos Examined

Crowdsourcing vs. professional translations

CLEF 2011 Amsterdam

Professional Translations

• Done on a per-word basis of 6-12 cents/word
• Average cost of US$49.65/translation
• Took an average of 4.7 business days to complete

Crowdsourcing Translations

• Average cost of US$2.15 per translation
• 1/23rd of the cost of a professional translation
• Took an average of 40 hours (1.6 days) to

complete

20

10/6/2011 11

Videos Examined Conclusion

CLEF 2011 Amsterdam

21

• Able to observe and quantify the advantage of

using video context clues over standard written translations alone

• Observe that some genres gain more from using

video context clues

• Observe that some languages gain more from

video context clues than others

• Crowdsourcing translations appear to be a cost-

effective way to obtain translations quickly

Videos Examined Thank you

CLEF 2011 Amsterdam

22

Questions?