Enhancing Gloss-Based Corpora with Facial Features Using Active - - PowerPoint PPT Presentation

Enhancing Gloss-Based Corpora with Facial Features Using Active Appearance Models

Christoph Schmidt, Oscar Koller, Hermann Ney 1 Thomas Hoyoux, Justus Piater 2

19.10.2013

1 Human Language Technology and Pattern Recognition Group

Computer Science Department RWTH Aachen University, Germany {surname}@i6.informatik.rwth-aachen.de

2 Intelligent and Interactive Systems

University of Innsbruck, Austria {firstname}.{surname}@uibk.ac.at

Schmidt, Koller Enhancing Gloss-Based Corpora 1 / 13 19.10.2013

SignSpeak

Sign Language Video Feature extraction (Image Analysis) Sign Language Recognition Sign Language Translation Spoken Language T ext System Architecture Scientific Understand

• f Sign Language

Radboud University Nijmegen

EUR

Market Research & Prototype Development Dissemination and Feedback from the Deaf Community

features glosses: MONDAY CHANGE SUN CLOUD spoken text: On Monday, the weather is changeable, partly sunny, partly cloudy.

◮ Goal: translate a sign language video into a spoken language text ◮ Project Duration: April 2009-March 2012

Schmidt, Koller Enhancing Gloss-Based Corpora 2 / 13 19.10.2013

Active Appearance Models

◮ track salient points on the face ◮ extract high-level facial features: ⊲ mouth vertical openness ⊲ mouth horizontal openness ⊲ lower lip to chin distance ⊲ upper lip to nose distance ⊲ left eyebrow state ⊲ right eyebrow state ⊲ gap between eyebrows ◮ necessary: labeled data

Schmidt, Koller Enhancing Gloss-Based Corpora 3 / 13 19.10.2013

RWTH-Phoenix-Weather Corpus

DGS German signers 7 editions 190 duration[h] 3.25 frames 293,077 sentences 2,711 glosses / words 17,744 33,190 vocabulary size 463 1,494 singletons 537 536 ◮ video-based, large vocabulary corpus ◮ weather forecasts from public TV news, interpreted into DGS ◮ annotation: glosses, time boundaries on gloss level ◮ focus on hand-based features Teaser: new version with 645 editions coming soon at LREC 2014 !

Schmidt, Koller Enhancing Gloss-Based Corpora 4 / 13 19.10.2013

Mouthing variants

ALPEN (“Alps”) BERG (“mountain”) ◮ Some signs only differ in mouthing / mouth gestures ◮ Annotation of RWTH-Phoenix-Weather focused on hand-based features ◮ Manual refinement of annotation time consuming ◮ Idea: automatic refinement using feature extraction and clustering ◮ Avatar animation: use refined annotation to animate mouthings / facial expressions

Schmidt, Koller Enhancing Gloss-Based Corpora 5 / 13 19.10.2013

Clustering Approach

◮ Cluster variants using AAM features ◮ Use the context of the spoken language to drive the clustering ◮ For avatar animation: select representative video ◮ Define distance between two videos: ◮ Train Hidden Markov Model on one video ◮ Calculate Viterbi path of second video

Schmidt, Koller Enhancing Gloss-Based Corpora 6 / 13 19.10.2013

Clustering Approach

◮ Align corpus ◮ Extract variants ◮ Cluster variants SL → Spoken

EVENING RIVER THREE MINUS SIX MOUNTAIN T

• night three degrees at the Oder, minus six degrees at the Alps .

EVENING_tonight RIVER_Oder MOUNTAIN_Alps EVENING_evening RIVER_Rhein MOUNTAIN_mountains MOUNTAIN_Alps MOUNTAIN_Eifel MOUNTAIN_Erzgebirge MOUNTAIN_Berge MOUNTAIN_Alps MOUNTAIN_Alps MOUNTAIN_Eifel MOUNTAIN_Eifel MOUNTAIN_Eifel MOUNTAIN_Erzgebirge MOUNTAIN_Erzgebirge MOUNTAIN_Berge MOUNTAIN_Berge MOUNTAIN_Berge Schmidt, Koller Enhancing Gloss-Based Corpora 7 / 13 19.10.2013

Clustering Approach

◮ Align corpus ◮ Extract variants ◮ Cluster variants SL → Spoken

EVENING RIVER THREE MINUS SIX MOUNTAIN T

• night three degrees at the Oder, minus six degrees at the Alps .

EVENING_tonight RIVER_Oder MOUNTAIN_Alps EVENING_evening RIVER_Rhein MOUNTAIN_mountains MOUNTAIN_Alps MOUNTAIN_Eifel MOUNTAIN_Erzgebirge MOUNTAIN_Berge MOUNTAIN_Alps MOUNTAIN_Alps MOUNTAIN_Eifel MOUNTAIN_Eifel MOUNTAIN_Eifel MOUNTAIN_Erzgebirge MOUNTAIN_Erzgebirge MOUNTAIN_Berge MOUNTAIN_Berge MOUNTAIN_Berge Schmidt, Koller Enhancing Gloss-Based Corpora 7 / 13 19.10.2013

Clustering Approach

◮ Align corpus ◮ Extract variants ◮ Cluster variants Spoken → SL

EVENING RIVER THREE MINUS SIX MOUNTAIN T

• night three degrees at the Oder, minus six degrees at the Alps .

EVENING_tonight RIVER_Oder MOUNTAIN_Alps EVENING_evening RIVER_Rhein MOUNTAIN_mountains STRONG_forceful m: strong m: strong m: strong m: strong m: forceful m: forceful m: forceful m: *puffed cheeks* m: *puffed cheeks* m: strong m: strong m: *puffed cheeks* Schmidt, Koller Enhancing Gloss-Based Corpora 7 / 13 19.10.2013

Clustering Approach

◮ Align corpus ◮ Extract variants ◮ Cluster variants Spoken → SL

EVENING RIVER THREE MINUS SIX MOUNTAIN T

• night three degrees at the Oder, minus six degrees at the Alps .

EVENING_tonight RIVER_Oder MOUNTAIN_Alps EVENING_evening RIVER_Rhein MOUNTAIN_mountains STRONG_forceful m: strong m: strong m: strong m: strong m: forceful m: forceful m: forceful m: *puffed cheeks* m: *puffed cheeks* m: strong m: strong m: *puffed cheeks* Schmidt, Koller Enhancing Gloss-Based Corpora 7 / 13 19.10.2013

Clustering Approach

◮ Align corpus ◮ Extract variants ◮ Cluster variants Spoken → SL

EVENING RIVER THREE MINUS SIX MOUNTAIN T

• night three degrees at the Oder, minus six degrees at the Alps .

EVENING_tonight RIVER_Oder MOUNTAIN_Alps EVENING_evening RIVER_Rhein MOUNTAIN_mountains STRONG_forceful m: strong

m: strong

m: strong m: strong m: forceful m: forceful m: forceful m: *puffed cheeks* m: *puffed cheeks* m: strong m: strong m: *puffed cheeks*

◮ Clustering algorithm: adaptive medoid-shift ◮ Select medoid of biggest cluster as representative video

Schmidt, Koller Enhancing Gloss-Based Corpora 7 / 13 19.10.2013

Experiments

◮ Annotate mouthings to evaluate clustering quality ◮ Select the most frequent glosses with more than one mouthing ◮ Select the most frequent contexts glosses 23 (gloss,translation) pairs 64 running glosses 640 GLOSS context MOUNTAIN Alps " mountain RIVER Rhine " Oder RAIN rain " shower EVENING evening " night

Schmidt, Koller Enhancing Gloss-Based Corpora 8 / 13 19.10.2013

Clustering results

1 32 64 20 40 60 80 100 Precision [%] (gloss,translation) avg:65.3% 1 32 64 20 40 60 80 100 Recall [%] (gloss,translation) avg:82.6% 1 32 64 20 40 60 80 100 F−Measure [%] avg:67.8% (gloss,translation)

◮ Precision: only same mouthings are in same cluster ◮ Recall: only different mouthings are in different clusters ◮ F-Measure: geometric mean of precision and recall

Schmidt, Koller Enhancing Gloss-Based Corpora 9 / 13 19.10.2013

Clustering results: biggest cluster

1 32 64 20 40 60 80 100 Accuracy [%] (gloss,translation) avg:78.4%

◮ Accuracy: medoid has same mouthing as other cluster members ◮ The overall algorithm achieves accuracy of 78.4%

Schmidt, Koller Enhancing Gloss-Based Corpora 10 / 13 19.10.2013

Clustering results: Examples

◮ left video: (MOUNTAIN,Allgaeu) ◮ right video: medoid of biggest cluster ◮ Algorithm can recognize same mouthing even among different signers

Schmidt, Koller Enhancing Gloss-Based Corpora 11 / 13 19.10.2013

Conclusion / Outlook

Conclusions: ◮ Clustering algorithm to detect variants in facial features ◮ Select representative video for avatar animation ◮ Achieves high accuracy Outlook: ◮ improve low-level features: histogram of mouth area ◮ improve high-level features: HMM → visemes ◮ apply method beyond mouthings: facial expressions, head shake, etc.

Schmidt, Koller Enhancing Gloss-Based Corpora 12 / 13 19.10.2013

Thank you for your attention

Christoph Schmidt

schmidt@i6.informatik.rwth-aachen.de http://www-i6.informatik.rwth-aachen.de/

Schmidt, Koller Enhancing Gloss-Based Corpora 13 / 13 19.10.2013

Appendix: Annotated glosses

GLOSS BIT NORTH BUT NOW CALAMITY RAIN CAN RIVER COLD SKY COURSE SNOW DRY SOUTH ESPECIALLY STRONG EVENING SUN HIGH TEMPERATURE MORE WIND MOUNTAIN GLOSS ABEND MEHR ABER NORD BERG REGEN BESONDERS SCHNEE BISSCHEN SONNE FLUSS STARK GEWITTER SUED HIMMEL TEMPERATUR HOCH TROCKEN JETZT VERLAUF KALT WIND KOENNEN

Schmidt, Koller Enhancing Gloss-Based Corpora 14 / 13 19.10.2013

Appendix: Cluster Evaluation

Precision = T P T P + F P , Recall = T P T P + F N , F-Measure = 2P R P + R ◮ True Positive: same mouthings is in same cluster ◮ True Negative: different mouthings is in different cluster ◮ False Positive: different mouthings are in same cluster ◮ False Negative: same mouthings are in different cluster

Schmidt, Koller Enhancing Gloss-Based Corpora 15 / 13 19.10.2013