Using multimodal speech production data to evaluate articulatory - PowerPoint PPT Presentation

Data Registration Animation Evaluation Using multimodal speech production data to evaluate articulatory animation for audiovisual speech synthesis Ingmar Steiner Korin Richmond Slim Ouni N I V E U R S E I H T T Y O H F G R E U D B I N University College Dublin CSTR LORIA & Trinity College Dublin University of Edinburgh Universit´ e de Lorraine Vienna, September 21, 2012

Data Registration Animation Evaluation Motivation Data-driven animation for speech articulators

Data Registration Animation Evaluation Motivation Data-driven animation for speech articulators within the vocal tract

Data Registration Animation Evaluation The mngu0 Corpus Multimodal speech corpus • one male speaker of British English • electromagnetic articulography (EMA) • magnetic resonance imaging (MRI) • dental cast scans http://mngu0.org/

Data Registration Animation Evaluation Electromagnetic articulography

Data Registration Animation Evaluation MRI volumetric vocal tract imaging

Data Registration Animation Evaluation MRI manual regions of interest (ROIs)

Data Registration Animation Evaluation MRI isosurfaces within ROIs

Data Registration Animation Evaluation Dental scans vertex count 927 282 (maxilla), 836 892 (mandible)

Data Registration Animation Evaluation Dental scans deduplication: vertex count 154 549 (maxilla), 139 484 (mandible)

Data Registration Animation Evaluation Dental scans decimate (5 %)

Data Registration Animation Evaluation Dental scans vertex count 7729 (maxilla), 6976 (mandible)

Data Registration Animation Evaluation Palate contour

Data Registration Animation Evaluation Palate contour

Data Registration Animation Evaluation Model rigging EMA motion capture data

Data Registration Animation Evaluation Model rigging maxilla/mandible track ref/jaw coils

Data Registration Animation Evaluation Tongue mesh retopology crude isosurface

Data Registration Animation Evaluation Tongue mesh retopology tesselation from MRI voxels

Data Registration Animation Evaluation Tongue mesh retopology simple cage

Data Registration Animation Evaluation Tongue mesh retopology “shrinkwrapped” to isosurface

Data Registration Animation Evaluation Tongue mesh retopology Catmull-Clark subdivision

Data Registration Animation Evaluation Tongue mesh retopology smooth, tongue-shaped mesh with simple topology

Data Registration Animation Evaluation Tongue rigging static mesh

Data Registration Animation Evaluation Tongue rigging spline (NURBS path)

Data Registration Animation Evaluation Tongue rigging modified by hooks tracking tongue coils

Data Registration Animation Evaluation Tongue rigging armature follows spline through inverse kinematics (IK)

Data Registration Animation Evaluation Tongue rigging tongue mesh deformed by armature

Data Registration Animation Evaluation Animation

Data Registration Animation Evaluation Animation

Data Registration Animation Evaluation Animation

Data Registration Animation Evaluation Animation

Data Registration Animation Evaluation Vertex tracking T1 T2 T3 6 4 2 x 0 -2 position (mm) 60 trajectory 50 40 EMA y 30 vertex 20 5 0 z -5 -10 0 1 2 3 0 1 2 3 0 1 2 3 time (s)

Data Registration Animation Evaluation Vertex tracking T1 T2 T3 6 4 2 x r=0.98 r=0.92 r=0.89 0 -2 position (mm) 60 trajectory 50 r=0.99 r=0.98 r=0.96 40 EMA y 30 vertex 20 5 r=0.99 r=0.92 r=0.91 0 z -5 -10 0 1 2 3 0 1 2 3 0 1 2 3 time (s)

Data Registration Animation Evaluation Conclusion Skeletal animation of articulatory movements from speech production data seems promising, but depends on • model topology • data quality • registration (incl. posture effect)