Advanced Animatronics Voice and Jaws v1.1 NordicFuzzCon – 20/02/2020 Floere T. Pillowcase, Devourer of Automobiles floere@robocow.be
What is this Talk About ? ● An overview of the State of the Art of moving jaws and voice projection ● Why I think their performance is ‘meh’ ● My research into a self-contained, real-time, speech expression mimicking character with a clear voice ● All the good ideas that weren’t... 2 / 64
Content ● The Goal ● State of the Art ● Why Moving Jaws Fail ● Mapping Human Speech to a Character ● Dealing with Speech in the Real World ● Jaw Motion Capture ● Voice Projection ● Putting it all together 3 / 64
Goal: Puppet Without Strings ● Your character driven by your acting ● Clear voice projection ● Live audience interaction ● Everything self- contained in the Lip-syncing with puppet mask (manual actuated) costume Radula Castion – Zuzu’s White Rabbit https://www.youtube.com/watch?v=b2pDuWh3ik8 ● Comfortable ● Affordable 4 / 64
Low Integration Complexity ● Easy enough to implement by hobbyists ● Not a movie-grade animatronic with 30+ servos and a head full of gears ● Simple mechanisms must suffice – Off-the-shelf parts – 3D printable 5 / 64 Gustav Hoegen
The Big Challenge ● Motion must be psychologically correct, not necessarily physiologically correct! ● A big, flappy mouth on a fuzzy critter is not exactly real… ● Uncanny valley helps Wikipedia - Uncanny Valley Conjecture (Mori 1970) → stay non-human! ● Not conveying the motion, but conveying the emotion ! 6 / 64
Content ● The Goal ● State of the Art ● Why Moving Jaws Fail ● Mapping Human Speech to a Character ● Dealing with Speech in the Real World ● Jaw Motion Capture ● Voice Projection ● Putting it all together 7 / 64
Let’s Watch Some Videos... All of these are live performances by the costume actor themselves ● (no lip-syncing or over-dubbing) Professional ● Katey McGregor – Talking Mickey Mouse – https://www.youtube.com/watch?v=762-tHwnAHg Mascot – Animatronic Mascots – https://www.youtube.com/watch?v=Ve3vuxII6Dc Lunaspuppets - Human-Size Animatronic Robotic Talking Donkey Puppet – https://www.youtube.com/watch?v=Cv5yAfHWEY4 Furry Fandom ● Bake Me Up Buttercup – How to Measure Flour Correctly – https://www.youtube.com/watch?v=YBkT5woqmAY Beautyofthe Bass – Speaker Costume Talks Live! V3 – https://www.youtube.com/watch?v=UWOWqe1kP7U ‿ DRAGON =^ ^= - Howwwwwwdy folks and welcome to Monday – Twitter: @GRNdragon0 8 / 64
It’s a Bit of a Mess, isn’t It? ● Professional work – Limited, static articulation (blinks + simple mouth) – Good voice quality ● ...is not actually the case! ● Often a remote voice actor involved ● Often pre-recorded phrases (semi-scripted) – Most costumes are actually puppets , controlled by the actor’s hand/chin/tongue, or a remote operator – Let’s have a look at this… The Character Academy – How Disney Characters Blink https://www.youtube.com/watch?v=YRDBFc-TrtM 9 / 64
It’s a Bit of a Mess, isn’t It? ● Amateur work is actually better in some ways – Articulated jaws can work (but often don’t) ● But it does not look like real speech! ● Good fit = uncomfortable to use for long – Voice is dull in real life ● YouTube videos use internal microphones ● Beautyofthe Bass is about the best one for live voice projection ● There are cosplayers who use the “TC Helicon Perform V” for voice projection, which works well (but bulky system) 10 / 64
Why is the Tech So Basic? ● There are many practicalities for the big boys that limit scope (getting the character voices right, consistency with many actors per costume, training requirements, etc …) ● The main reason, I think, is because it is actually a hard problem to solve in practice ● It would take a lot of money, or a motivated idiot with a PhD... 11 / 64
Content ● The Goal ● State of the Art ● Why Moving Jaws Fail ● Mapping Human Speech to a Character ● Dealing with Speech in the Real World ● Jaw Motion Capture ● Voice Projection ● Putting it all together 12 / 64
Why Moving Jaws Fail for Speech ● Fundamentally: moving jaws do not work well while speaking because normal speech does not use much jaw motion ● Any slop in the mechanism dulls jaw motion ● Some performers can make their jaw work – Speaking with exaggerated jaw motion – E.g.: Buttercup and NIIC do this well ● Still does not feel right… (hint: visemes) 13 / 64
What the Science Says... ● There are two sets of muscles in the jaw: – Big and very powerful ones for chewing and large jaw motions. These are slow ! – Little , fast ones for speech – The big ones disengage when speaking ● Jaw motion during speech is usually small: – Under ~0.3 cm pronouncing / ta / and / te / Ostry and Flanagan, 1989 ● Some sounds (eg: vowels) can have large motion: – Under ~2.5 cm pronouncing / a / Vatikiotis-Bateson and Ostry, 1995 14 / 64
What the Science Says... Sensor attached to the chin, just posterior to the mental notch. “Human Jaw Movement in Mastication and Speech”, D.J. Ostry and J.R. Flanagan, 15 / 64 Archs. Oral Biol. Vol. 34, No. 9, pp. 685-693, 1989
What the Science Says... Marker 4 cm from lower incisors, ~on the midsagittal plane. “An Analysis of the Dimensianality of Jaw Motion in Speech”, E. Vatikiotis-Bateson 16 / 64 and D.J. Ostry, Journal of Phonetics, Vol. 23, pp. 101-117, 1995
Content ● The Goal ● State of the Art ● Why Moving Jaws Fail ● Mapping Human Speech to a Character ● Dealing with Speech in the Real World ● Jaw Motion Capture ● Voice Projection ● Putting it all together 17 / 64
First, a Little... 18 / 64
How Speech is Produced K. Duh, M. Lloyd, M. Smiley Haskins Laboratories 19 / 64 gosh.nhs.uk
How Speech is Produced Jörgen Ahlberg – Source-Filter Model of Speech Production 20 / 64
How Speech is Produced ● You sound like a fat bee inside! – Voiced speech starts from glottal impulses – Bzz! Bzzzzzz! – Recorded using a contact microphone – Is also why throat microphones sound iffy… 21 / 64
Phonemes vs Visemes Animators learn that much of ● visible speech is lip motion They use only a few visemes ● – Many speech sounds (phonemes) look alike – Eg: to a lip reader “elephant juice” = “I love you” Thus: we can simplify a lot ● Can we get phonemes from ● speech? – A very hard problem 22 / 64 – Key to speech recognition
Mouth Shape from Sound? ● Look at the visemes and try the utterances – Voiced or louder → mouth more open – Nasal or unvoiced → mouth more closed ● Try: “mama” “is” “na” ● Not perfect, but should be good enough for a simple jaw 23 / 64 Wolf Paulus – Viseme Model with 12 Mouth Shapes
How We’re Going to Do It ● Jaw sensor ● Key idea: rough visemes – Chin motion (slow) – Estimate mouth state from jaw + lips – Measured from jaw – No actual phoneme – Includes static poses detection ● Lip “sensor” (or/na mic) – Don’t need perfection – Lip motion (fast) – Estimated from speech Jaw sensor – No action when silent Mouth Jaw Est. Servos Lip Speech “sensor” Analysis 24 / 64
Voicedness + Nasalence ● Voicing detection – Voiced, unvoiced, or silence? – How much energy? – Can we do this implicitly? “A pattern recognition approach to voiced-unvoiced-silence classification with applications to speech recognition,” Bishnu S. Atal, Lawrence R. Rabiner, 1976. 25 / 64
Voicedness + Nasalence ● Nasalence – How nasal is voiced speech? – Have done original research on sensors… – But can’t use any of that Donald Derrick – nasalence of na work here! ● Too bulky due to underlying principle ● Had to measure airflow + speech ● Not required here! 26 / 64
Bringing it All Together ● Jaw activity gets us the “wide open” visemes, as well as silent + static mouth motions ● Speech activity opens the lips ● Unvoiced speech and high nasalence counter-act the lip opening ● Thus: voice signal adds the lost small (fast) lips motion to the large (slow) jaw motion – Lips can be separate or added to jaw motion 27 / 64
Bringing it All Together ● Mechanism – Jaw → 1 servo On jaw hinge – Lips → 1/2 servos (opt.) On lip actuation wires ● Sensors Eva Taylor – Animatronic Alien https://makezine.com/2014/10/27/the-making-of-an-animatronic-alien/ – Two microphones (mouth + nose) – Jaw motion sensor 28 / 64
Mechanisms skud duncan – Animatronic Jaw Test Winter Snowmew - “Couple of my followers have been curious https://www.youtube.com/watch?v=15IVl1VYdSk about the weird snout. Here is the snarl and mouth mechanics.” Tioh ● http://www.tioh.de/ Radula Castion ● https://radulacastion.wixsite.com/radulacastion “Animatronic Character Creation – Organic Mechanics I & II,” ● Rick Lazzarini, Stan Winston School of Character Arts 29 / 64
How Good is “Simple”? ● We gain a lot with only jaw, or jaw + simple lips (1 – 3 servos) ● Full expression of movie-grade animatronic mouth would require many more servos and much more complex motion capture system – This is not the point of this project – Afforability and “bang for the buck” is key 30 / 64
Recommend
More recommend
