Using multimodal speech production data to evaluate articulatory - - PowerPoint PPT Presentation

▶
using multimodal speech production data to evaluate
SMART_READER_LITE
LIVE PREVIEW

Using multimodal speech production data to evaluate articulatory - - PowerPoint PPT Presentation

Apr 03, 2024 46 likes •389 views

Data Registration Animation Evaluation Using multimodal speech production data to evaluate articulatory animation for audiovisual speech synthesis Ingmar Steiner Korin Richmond Slim Ouni N I V E U R S E I H T T Y O H F G R

slide-1
SLIDE 1

Data Registration Animation Evaluation

Using multimodal speech production data to evaluate articulatory animation for audiovisual speech synthesis

Ingmar Steiner Korin Richmond Slim Ouni

T H E U N I V E R S I T Y O F E D I N B U R G H

University College Dublin CSTR LORIA & Trinity College Dublin University of Edinburgh Universit´ e de Lorraine

Vienna, September 21, 2012

slide-2
SLIDE 2

Data Registration Animation Evaluation

Motivation

Data-driven animation for speech articulators

slide-3
SLIDE 3

Data Registration Animation Evaluation

Motivation

Data-driven animation for speech articulators within the vocal tract

slide-4
SLIDE 4

Data Registration Animation Evaluation

The mngu0 Corpus

Multimodal speech corpus

  • one male speaker of British English
  • electromagnetic articulography (EMA)
  • magnetic resonance imaging (MRI)
  • dental cast scans

http://mngu0.org/

slide-5
SLIDE 5

Data Registration Animation Evaluation

Electromagnetic articulography

slide-6
SLIDE 6

Data Registration Animation Evaluation

MRI

volumetric vocal tract imaging

slide-7
SLIDE 7

Data Registration Animation Evaluation

MRI

manual regions of interest (ROIs)

slide-8
SLIDE 8

Data Registration Animation Evaluation

MRI

isosurfaces within ROIs

slide-9
SLIDE 9

Data Registration Animation Evaluation

Dental scans

vertex count 927 282 (maxilla), 836 892 (mandible)

slide-10
SLIDE 10

Data Registration Animation Evaluation

Dental scans

deduplication: vertex count 154 549 (maxilla), 139 484 (mandible)

slide-11
SLIDE 11

Data Registration Animation Evaluation

Dental scans

decimate (5 %)

slide-12
SLIDE 12

Data Registration Animation Evaluation

Dental scans

vertex count 7729 (maxilla), 6976 (mandible)

slide-13
SLIDE 13

Data Registration Animation Evaluation

Palate contour

slide-14
SLIDE 14

Data Registration Animation Evaluation

Palate contour

slide-15
SLIDE 15

Data Registration Animation Evaluation

Model rigging

EMA motion capture data

slide-16
SLIDE 16

Data Registration Animation Evaluation

Model rigging

maxilla/mandible track ref/jaw coils

slide-17
SLIDE 17

Data Registration Animation Evaluation

Tongue mesh retopology

crude isosurface

slide-18
SLIDE 18

Data Registration Animation Evaluation

Tongue mesh retopology

tesselation from MRI voxels

slide-19
SLIDE 19

Data Registration Animation Evaluation

Tongue mesh retopology

simple cage

slide-20
SLIDE 20

Data Registration Animation Evaluation

Tongue mesh retopology

“shrinkwrapped” to isosurface

slide-21
SLIDE 21

Data Registration Animation Evaluation

Tongue mesh retopology

Catmull-Clark subdivision

slide-22
SLIDE 22

Data Registration Animation Evaluation

Tongue mesh retopology

smooth, tongue-shaped mesh with simple topology

slide-23
SLIDE 23

Data Registration Animation Evaluation

Tongue rigging

static mesh

slide-24
SLIDE 24

Data Registration Animation Evaluation

Tongue rigging

spline (NURBS path)

slide-25
SLIDE 25

Data Registration Animation Evaluation

Tongue rigging

modified by hooks tracking tongue coils

slide-26
SLIDE 26

Data Registration Animation Evaluation

Tongue rigging

armature follows spline through inverse kinematics (IK)

slide-27
SLIDE 27

Data Registration Animation Evaluation

Tongue rigging

tongue mesh deformed by armature

slide-28
SLIDE 28

Data Registration Animation Evaluation

Animation

slide-29
SLIDE 29

Data Registration Animation Evaluation

Animation

slide-30
SLIDE 30

Data Registration Animation Evaluation

Animation

slide-31
SLIDE 31

Data Registration Animation Evaluation

Animation

slide-32
SLIDE 32

Data Registration Animation Evaluation

Vertex tracking

T1 T2 T3

  • 2

2 4 6 20 30 40 50 60

  • 10
  • 5

5 x y z 1 2 3 1 2 3 1 2 3

time (s) position (mm)

trajectory EMA vertex

slide-33
SLIDE 33

Data Registration Animation Evaluation

Vertex tracking

T1 T2 T3

  • 2

2 4 6 20 30 40 50 60

  • 10
  • 5

5

r=0.98 r=0.99 r=0.99 r=0.92 r=0.98 r=0.92 r=0.89 r=0.96 r=0.91

x y z 1 2 3 1 2 3 1 2 3

time (s) position (mm)

trajectory EMA vertex

slide-34
SLIDE 34

Data Registration Animation Evaluation

Conclusion

Skeletal animation of articulatory movements from speech production data seems promising, but depends on

  • model topology
  • data quality
  • registration (incl. posture effect)