Paper Summaries Any takers? Sound and Animation This week is the - PDF document

Paper Summaries • Any takers? Sound and Animation • This week is the last week for paper summaries. Annoucements Announcements • AniFest 04 • Want to learn Maya? – Western Connecticut State University – 2001-753 – http://149.152.225.94/festival04.html – 3D Modeling in Maya – Deadline for submission: Feb 28 th – Designed for gamers and non-design majors – See me for more details – Contact: Marla Schweppe • mkspph@rit.edu Grad Report Projects • Presentations: • Final Reports – February 11 th – Next Wednesday. – Note that final reports / code are SEPARATE grading components • Written reports – Due by Feb 18 th – last class. – Final reports/code are due on the last day of class (Feb 18 th ) 1

Projects Assignments • Final Report • Assignment #1 – Textual description of your system – Submitted and graded – Sections • Assignment #2 • Problem/Project Description • Approach – Grace period ends today • Implementation – Overall System Architecture. • Assignment #3 – Overall Program Architecture – Due Feb 11 th (next Wednesday) – Description of major data structures / objects • Results / User Documentation • Future Enhancements • Appendix -- All Code listings Plan for today About Monday’s class…. • Today: Sound and Animation • Sound and Animation • Monday: – Project day • No lecture • Will be in office • Wednesday: Grad Reports • Following Week: – Project Presentations Motivation Films Motivational Film • Animations by Wayne Lytle • More Bells and Whistles (1990) – Visualization Guru at Cornell Theory Center – Lytle wrote the the code for each band member – Quit to start Animusic in 1995 – Motion is MIDI controlled – First of several Animusic pieces to be shown at SIGGRAPH 2

Motivational Film Motivational Film • Pipe Dream (2001) • Train Wreck (2003) – Animusic • Martin Burolla – Can’t See too much Animusic • From last year’s animation class – Sound drives motion Sound and Animation Sound • Issues in Sound and Animation • What is sound? – Sound Generation – From webster.com • What do we play? • mechanical radiant energy that is transmitted by longitudinal pressure waves in a material medium – Sound Synchronization (as air) and is the objective cause of hearing • When do we play? – Spatial Sound • Where do we play Sound Remember this? • What is sound? • Spatial vs frequency domains – Sound can be described as a 1 dimensional – Most well behaved functions can be described signal in time as a sum of sin waves (possibly offset) at various frequencies sound = f(t) – Describing a function by the contribution (and offset) at each frequency is describing the function in the frequency domain 3

Sound Sound • A mathematical description of an audio signal: ∞ = ∑ π ω + φ f ( t ) A sin( 2 t ) i i i = i 0 Contribution/amplitude frequency phase Foley/VanDam Sound: Loudness Sound: Pitch • Looking at sound in the temporal domain • Looking at sound in the frequency domain. – Sound can be described as a 1 dimensional – Humans “hear” sounds because of periodicities signal in time in the audio signal. – Humans perceive frequency as the sensation of pitch. – Signal values represent amplitude. – Humans can perceive pitches due to periodicities ranging from 20 – 20000 – We perceive the effect of amplitude as vibrations / sec (Hz). loudness. Sound: Pitch Sound: Timbre • Tone quality of a sound • Remember our discussion of CD audio • Formally defined as – sampling rate of 44,100 samples/sec – ∆ = 1 sample every 2.26x10 -5 seconds – Characteristic of sound not due to amplitude and pitch. – CDs can accurately reproduce sounds with • Also defined frequencies as high as 22,050 Hz. – Quality of tone that distinguishes between musical instruments – Sound shape 4

Sound: Timbre Sound: Summary • Timbre is the perception of the “spectral Physical Perceptual makeup” of a signal. Characteristic Characteristic – Adding non-fundamental frequency to the Amplitude Loudness signal. Frequency Pitch – Another annoying audio applet – Timbre Spectral “shape” Timbre Sound Generation Sound Generation • So how does one generate sound for • When talking about digital (sampled sound) animation? – The process of digitizing is called pulse code modulation (PCM). – Easiest means – PCM == sampled sound • Recording / Sampling -- Still the primary means for sound generation in the film industry • WAV • Using sampled sound – Still the primary means for • AIFF sound use in games. • MP3 (compressed PCM) Sound Generation Sound Generation • Subtractive Synthesis • Additive Synthesis ∞ ∞ = ∑ = ∑ π ω + φ π ω + φ f ( t ) A sin( 2 t ) f ( t ) A sin( 2 t ) i i i i i i = = i 0 i 0 • Define values for A i , ω i , and φ i • Start with noise (equal energies at all frequencies) • Calculate sin and add • Alternately, do in the addition in frequency space. • Subtract contribution of frequencies from noise. 5

Sound Generation Sound Synchronization • Granular Synthesis • Sound must be synchronized to the motion – Like particle system – Methods: • Motion driving sound – Defining Sound events – Deriving timbre from motion • Sound driving motion – Combine a multitude of sound “grains” into a sound events – Questions Sound Synchronization Sound Synchronization • Generating sound from physical simulation • Sound driving motion – Video examples – MIDI • Designed as a communication mode between sythesizers, samplers, instruments, computers • Sound events – Pitch – Devices • Used by Animusic in creating their videos Spatial Sound 3D Sound • Sounds (and listeners) have spatial positions • Making sounds appear as if they are emitted – 3D sound from a given position accounting for • Making sounds appear as if they are emitted from a listener position given position accounting for listener position – Head related transfer functions (HRTF) – Reverberation – Audio cubes / surround sound • Filtering of sound based on reflection off of environment • Strategic placing of speakers – Doppler Effect • Change in pitch due to moving objects 6

3D Sound: HRTF 3D Sound: HRTF • a description of all the physical cues of sound localisation. – Implemented as filters – function of four variables: ie three space coordinates and frequency. – Determined by measurement Anderson/Casey 3D Sound: HRTF 3D Sound: reverberation • Like light, sound can be seen as traveling in 3D environment in rays. • Unlike light, sound travels much slower – Speed of sound: – Speed of light Anderson/Casey 3D Sound: reverberation 3D Sound: reverberation • Reverberant sound is the • Examples collection of all the – From BKL Consultants Ltd. reflected sounds in an (http://www.bkla.com/reverb.htm) enclosed space • No reverb • Acoustics • Reverb Time = time • 0.8 sec reverb time required for sound to • 1.5 sec reverb time decay one millionth of the • 5.0 sec reverb time original power 7

3D Sound: Doppler effect 3D Sound: Doppler effect • Non-annoying applet Sound: Putting It all Together Sound: Putting It all Together • Sound Rendering Video Examples Takala/Hahn Remember CGII: Procedural Sound: Putting It all Together Shading • Shade Trees [Cook84] • Questions? – Shading calculated by combining basic functional operations. – Operations are organized in a tree. • Nodes - Operations • Children - operands • Break! – Result of shade tree evaluation is a color – Equiv to parse tree (compiler design) – Basis of Renderman shading language. 8

Remember CGII: Procedural Shading Remember CGII: Procedural Shading • Shade Trees - example…copper • Basic ideas behind shade trees: – Describe textures / shading functionally – Using Parameters from 3D world • Can we use a similar model for sound? [Cook84] Timbre Trees Timbre trees • Functional sound synthesis – Sound related functions • Periodic functions • Convolution • Noise • Filtering – Nodes for animation, 3d parameters Hahn/Geigel. Et al Timbre trees Timbre trees • Nodes could also be used to simulate: – Reverberation – Delay – Spatial Sound Hahn/Geigel. Et al 9

Timbre Trees Genetic Texture • What we failed to realize – Functional sound, unlike functional textures, (sin (+ (- (grad-direction (blur (if (hsv- was far from novel… to-rgb (warped-color-noise #(0.57 0.73 0.92) (/ 1.85 (warped-color-noise x y – Quite popular in the Computer Music circles 0.02 3.08)) 0.11 2.4)) #(0.54 0.73 0.59) • Nyquist -- CMU #(1.06 0.82 0.06)) 3.1) 1.46 5.9) (hsv- to-rgb (warped-color-noise y (/ 4.5 • csound – MIT (basis of MPEG-4 Structured Audio) (warped-color-noise y (/ x y) 2.4 2.4)) – However… 0.02 2.4))) x)) [Sims91] Genetic Sound Timbre Tree • Since Timbre trees were nothing more than • Video examples functional description of sound (using LISP expressions) – Experimentation with genetic manipulation was natural Good news about this research Bad news about this research • Sound now integrated as part of rendering • Sound effects for motion pictures is still pipeline done using foley artists – DirectSound – VRML2.0 – openAL 10

Questions • Next time – No lecture 11