Advanced Animatronics Voice and Jaws v1.0 Flüüfgf – 22/11/2019 Floere T. Pillowbeaver, Devourer of Nuclear Submarines fmoere@robocow.be
What is this Talk About ? ● An overview of the State of the Art of moving jaws and voice projection ● Why I think their performance is ‘meh’ ● My research into a self-contained, real-time, speech expression mimicking character with a clear voice ● All the good ideas that weren’t... 2 / 60
Content ● The Goal ● State of the Art ● Why Moving Jaws Fail ● Mapping Human Speech to a Character ● Dealing with Speech in the Real World ● Jaw Motion Capture ● Voice Projection ● Putting it all together 3 / 60
Goal: Puppet Without Strings ● Your character driven by your acting ● Clear voice projection ● Live audience interaction ● Everything self- contained in the costume Lip-syncing with puppet mask (manual actuated) ● Comfortable Radula Castion – Zuzu’s White Rabbit https://www.youtube.com/watch?v=b2pDuWh3ik8 ● Afgordable 4 / 60
Low Integration Complexity ● Easy enough to implement by hobbyists ● Not a movie-grade animatronic with 30+ servos and a head full of gears ● Simple mechanisms must suffjce – Ofg-the-shelf parts – 3D printable 5 / 60 Gustav Hoegen
The Big Challenge ● Motion must be psychologically correct, not necessarily physiologically correct! ● A big, fmappy mouth on a fuzzy critter is not exactly real… Wikipedia - Uncanny Valley Conjecture (Mori 1970) ● Uncanny valley helps → stay non-human! 6 / 60
Content ● The Goal ● State of the Art ● Why Moving Jaws Fail ● Mapping Human Speech to a Character ● Dealing with Speech in the Real World ● Jaw Motion Capture ● Voice Projection ● Putting it all together 7 / 60
Let’s Watch Some Videos... All of these are live performances by the costume actor themselves ● (no lip-syncing or over-dubbing) Professional ● Katey McGregor – T alking Mickey Mouse – https://www.youtube.com/watch?v=762-tHwnAHg Mascot – Animatronic Mascots – https://www.youtube.com/watch?v=Ve3vuxII6Dc Lunaspuppets - Human-Size Animatronic Robotic T alking Donkey Puppet – https://www.youtube.com/watch?v=Cv5yAfHWEY4 Furry Fandom ● Bake Me Up Buttercup – How to Measure Flour Correctly – https://www.youtube.com/watch?v=YBkT5woqmAY Beautyofthe Bass – Speaker Costume T alks Live! V3 – https://www.youtube.com/watch?v=UWOWqe1kP7U DRAGON =^‿^= - Howwwwwwdy folks and welcome to Monday – T witter: @GRNdragon0 8 / 60
It’s a Bit of a Mess, isn’t It? ● Professional work – Limited, static articulation (blinks + simple mouth) – Good voice quality ● ...is not actually the case! ● Often a remote voice actor involved ● Often pre-recorded phrases (semi-scripted) – Most costumes are actually puppets , controlled by the actor’s hand/chin/tongue, or a remote operator – Let’s have a look at this… The Character Academy – How Disney Characters Blink https://www.youtube.com/watch?v=YRDBFc-TrtM 9 / 60
It’s a Bit of a Mess, isn’t It? ● Amateur work is actually better in some ways – Articulated jaws can work (but often don’t) ● But it does not look like real speech! ● Good fjt = uncomfortable to use for long – Voice is dull in real life ● YouTube videos use internal microphones ● Beautyofthe Bass is about the best one for live voice projection ● There are cosplayers who use the “TC Helicon Perform V” for voice projection, which works well (but bulky system) 10 / 60
Why is the Tech So Basic? ● There are many practicalities for the big boys that limit scope (getting the character voices right, consistency with many actors per costume, training requirements, etc …) ● The main reason, I think, is because it is actually a hard problem to solve in practice ● It would take a lot of money, or a motivated idiot with a PhD... 11 / 60
Content ● The Goal ● State of the Art ● Why Moving Jaws Fail ● Mapping Human Speech to a Character ● Dealing with Speech in the Real World ● Jaw Motion Capture ● Voice Projection ● Putting it all together 12 / 60
Why Moving Jaws Fail for Speech ● Fundamentally: moving jaws do not work well while speaking because normal speech does not use much jaw motion ● Any slop in the mechanism dulls jaw motion ● Some performers can make their jaw work – Speaking with exaggerated jaw motion – E.g.: Buttercup and NIIC do this well ● Still does not feel right… (hint: visemes) 13 / 60
What the Science Says... ● There are two sets of muscles in the jaw: – Big and very powerful ones for chewing and large jaw motions. These are slow ! – Little , fast ones for speech – The big ones disengage when speaking ● Jaw motion during speech is usually small: – Under ~0.3 cm pronouncing / ta / and / te / Ostry and Flanagan, 1989 ● Some sounds (eg: vowels) can have large motion: – Under ~2.5 cm pronouncing / a / Vatikiotis-Bateson and Ostry, 1995 14 / 60
What the Science Says... Sensor attached to the chin, just posterior to the mental notch. “Human Jaw Movement in Mastication and Speech”, D.J. Ostry and J.R. Flanagan, 15 / 60 Archs. Oral Biol. Vol. 34, No. 9, pp. 685-693, 1989
What the Science Says... Marker 4 cm from lower incisors, ~on the midsagittal plane. “An Analysis of the Dimensianality of Jaw Motion in Speech”, E. Vatikiotis-Bateson 16 / 60 and D.J. Ostry, Journal of Phonetics, Vol. 23, pp. 101-117, 1995
Content ● The Goal ● State of the Art ● Why Moving Jaws Fail ● Mapping Human Speech to a Character ● Dealing with Speech in the Real World ● Jaw Motion Capture ● Voice Projection ● Putting it all together 17 / 60
First, a Little... 18 / 60
How Speech is Produced K. Duh, M. Lloyd, M. Smiley Haskins Laboratories 19 / 60 gosh.nhs.uk
How Speech is Produced Jörgen Ahlberg – Source-Filter Model of Speech Production 20 / 60
Phonemes vs Visemes Animators learn that much of ● visible speech is lip motion They use only a few visemes ● – Many speech sounds (phonemes) look alike – Eg: to a lip reader “elephant juice” = “I love you” Thus: we can simplify a lot ● Can we get phonemes from ● speech? – A very hard problem – Key to speech recognition 21 / 60
Mouth Shape from Sound? ● Look at the visemes and try the utterances – Voiced or louder → mouth more open – Nasal or unvoiced → mouth more closed ● Try: “mama” “is” “na” ● Not perfect, but should be good enough for a simple jaw 22 / 60 Wolf Paulus – Viseme Model with 12 Mouth Shapes
How We’re Going to Do It ● Jaw sensor ● Key idea: rough visemes – Chin motion (slow) – Estimate mouth state from jaw + lips – Measured from jaw – No actual phoneme – Includes static poses detection ● Lip “sensor” (or/na mic) – Don’t need perfection – Lip motion (fast) – Estimated from Jaw sensor speech Mouth Jaw – No action when silent Est. Servos Lip Speech “sensor” Analysis 23 / 60
Voicedness + Nasalence ● Voicing detection – Voiced, unvoiced, or silence? – How much energy? “A pattern recognition approach to voiced-unvoiced-silence classification with applications to speech recognition,” Bishnu S. Atal, Lawrence R. Rabiner, 1976. ● Nasalence – How nasal is voiced speech? ● Have done original research on sensors 24 / 60 Donald Derrick – nasalence of na
Bringing it All Together ● Jaw activity gets us the “wide open” visemes, as well as silent + static mouth motions ● Speech activity opens the lips ● Unvoiced speech and high nasalence counter- act the lip opening ● Thus: voice signal adds the lost small (fast) lips motion to the large (slow) jaw motion – Lips can be separate or added to jaw motion 25 / 60
Bringing it All Together ● Mechanism – Jaw → 1 servo On jaw hinge – Lips → 1/2 servos (opt.) On lip actuation wires ● Sensors Eva Taylor – Animatronic Alien https://makezine.com/2014/10/27/the-making-of-an-animatronic-alien/ – T wo microphones (mouth + nose) – Jaw strap 26 / 60
Mechanisms skud duncan – Animatronic Jaw Test Winter Snowmew - “Couple of my followers have been curious https://www.youtube.com/watch?v=15IVl1VYdSk about the weird snout. Here is the snarl and mouth mechanics.” Tioh ● http://www.tioh.de/ Radula Castion ● https://radulacastion.wixsite.com/radulacastion “Animatronic Character Creation – Organic Mechanics I & II,” ● Rick Lazzarini, Stan Winston School of Character Arts 27 / 60
How Good is “Simple”? ● We gain a lot with only jaw, or jaw + simple lips (1 – 3 servos) ● Full expression of movie-grade animatronic mouth would require many more servos and much more complex motion capture system – This is not the point of this project – Afgorability and “bang for the buck” is key 28 / 60
Does Simple Lose Much? ● Let’s compare high- end animatronics to a well-done lip sync ● I think small errors in animation are working against it → uncanny valley Shanetheactor – MetroPCS Commercial https://www.youtube.com/watch?v=udlQ7SH_RtM ● Clearly: deminishing returns VS TheCharacterShop – TCSpolarbearWaldo.mov 29 / 60 https://www.youtube.com/watch?v=bFW2azvVEdI Radula Castion – Zuzu’s White Rabbit https://www.youtube.com/watch?v=b2pDuWh3ik8
Recommend
More recommend
