Towards an Intuitive Sign Language Animation Authoring Environment - - PowerPoint PPT Presentation

▶
towards an intuitive sign language animation authoring
SMART_READER_LITE
LIVE PREVIEW

Towards an Intuitive Sign Language Animation Authoring Environment - - PowerPoint PPT Presentation

Jun 07, 2023 472 likes •667 views

Towards an Intuitive Sign Language Animation Authoring Environment For the Deaf Alexis Heloir, Fabrizio Nunnari UVHC, LAMIH - Valenciennes, France DFKI-MMCI, SLSI group - Germany NLP: Parallel Corpora Aligned annotated videos Aligned Glosses

slide-1
SLIDE 1

Towards an Intuitive Sign Language Animation Authoring Environment For the Deaf

Alexis Heloir, Fabrizio Nunnari

UVHC, LAMIH - Valenciennes, France DFKI-MMCI, SLSI group - Germany

slide-2
SLIDE 2

Aligned Glosses

DictaSign Project (2009--2012)

NLP: Parallel Corpora

Aligned annotated videos

RWTH-Phoenix corpus (2010)

slide-3
SLIDE 3

NLP: Corpora

Motion capture

CUNY SL Corpus (2012) SignCom System (2012)

Hand-crafting

ATLAS (2009-- 2011)

slide-4
SLIDE 4

SL Corpora

  • Videos
  • Motion capture
  • Hand-crafted

animation

  • Manual annotation
slide-5
SLIDE 5

Video

+ Easy to use and accessible + High fidelity

  • Low semantics
slide-6
SLIDE 6

Motion Capture

+ High fidelity + 3D plus numerical values

  • Still Low semantics
  • Cumbersome and expensive
slide-7
SLIDE 7

Hand-crafted animation

+ More intelligible animation + Theory independent motion abstraction

  • Expensive
  • Requires both expert interpreters

and 3D animators

slide-8
SLIDE 8

Annotation

+ High semantic + Abstracted away

  • Requires expert annotators
  • Theory Dependent
slide-9
SLIDE 9

Synthetic signs

??? Hand-crafted animation

SL corpuses

Videos Motion Capture

A n n

  • t

a t i

  • n

s

(Sign Writing, HamNoSys)

Quantity of data Abstraction

slide-10
SLIDE 10

Synthetic signs

Monitored handcrafted avatar animation Hand-crafted animation

SL corpuses

Videos Motion Capture

A n n

  • t

a t i

  • n

s

(Sign Writing, HamNoSys)

Quantity of data Abstraction

NEW!

slide-11
SLIDE 11

Current SL production pipelines

Performance Capture Data Cleanup

Hired Performer / Interpreter

Mocap

  • perator

Sign animation Cleanup 3D

  • perator

Signary Video Capture Sign animation 3D operator & Interpreter

mocap → hand-craft →

slide-12
SLIDE 12

Production pipeline: to the user

Performance Capture Data Cleanup

Deaf user

Mocap

  • perator

Sign animation Cleanup 3D

  • perator

Signary Video Capture Sign animation 3D operator & Interpreter

mocap → hand-craft →

slide-13
SLIDE 13

Production pipeline: no complex 3D authoring

Performance Capture Data Cleanup

Deaf user

Mocap

  • perator

Sign animation Cleanup 3D

  • perator

Signary

mocap → hand-craft →

slide-14
SLIDE 14

Production pipeline: NUIs and no specialized operators

Performance Capture Data Cleanup

Deaf user

Mocap

  • perator

Sign animation Cleanup 3D

  • perator

Signary

NUI !!!

slide-15
SLIDE 15

Production pipeline: possibility to refine captured signs

Performance Capture Data Cleanup

Deaf user

Sign animation Cleanup Signary

slide-16
SLIDE 16

New SL production pipelines

Performance Capture Data Cleanup

Deaf user

Manual Edit Signary

slide-17
SLIDE 17

Production pipeline at work

Video

slide-18
SLIDE 18

Future: Gathering users’ intention

  • LIS relevant info

○ Contacts ○ Time alignments ○ Gesture amplification ○ ...

Patterns extraction Edit history Manual Edit