Projects Presentations: Sound and Animation Dates: Week 9: Wed, - - PDF document

SLIDE 1

1

Sound and Animation

Projects

 Presentations:

 Dates:  Week 9: Wed, Feb 14  Week 10: Mon, Feb 19  Finals Week: Tues, Feb 27 (2:45-4:45)  15 minutes / presentation  Schedule now on Web  Please send me choice of time/day  ALL PRESENTATIONS HAVE BEEN SCHEDULED!!!  PRESENTATIONS START ON WEDNESDAY

Assignments



Assignment 1 -- Framework



Assignment 2 -- Keyframing



Most have been graded.



Assignment 3 -- Billiards



50% graded



Assignment 4 -- Group Motion



To be done Wednesday-Thursday



Assignment 5 - Walking



Due 2/16 (Friday)



NOTE: Dropbox close dates have been fixed.

Plan

 Today:

 Last lecture: Sound and Animation

 Wednesday

 Presentations -- Day 1

 Next Monday

 Presentations -- Day 2

 Next Wednesday

 NO CLASS

 Feb 27th (Finals week)

 Presentation -- Day 3  LAST DAY TO HAND IN STUFF.

Thinking About Spring?



Applications in Virtual Reality (4003-590-09 / 4005-769-09)



Virtual Theatre



A distributed computer system whereby performers, stage crew, and audience can be in physically separate places yet share in the same live theatrical performance.



Components



Torque Gaming Engine



moCap devices



Game networking



Details



4003-590-09 / 4005-769-09



TR 2-4 (ICL6) 

Please contact Joe Geigel (jmg@cs.rit.edu) for details

Thinking About Spring



Procedural Shading



The goal of this course is to introduce students to the architectures and mechanisms of procedural shading and to teach them how to use shaders effectively in creating stunning visual effects.



Components



Advanced RenderMan



Real time shaders (Cg)



Lecture + Lab/Studio



Actual shaders form artists specs.



Details



4003-590-01 / 4005-769-01



TR 4-6 (ICL6)



PRE-REQ: Computer Graphics II

SLIDE 2

2 Jobs, Jobs, Jobs

 Full time position  Intel

 GPU validation  Experience in 3D and/or hardware  May Graduation date  Several alum there

 E-mail me w/resume if interested.

Jobs, Jobs, Jobs

 Demiurge (Boston area)

 Game company  Interns, co-ops, full-time  Programmers who likes games  www.demiurgestudios.com

 Send resume to me, I will forward.

Jobs, Jobs, Jobs

 Possible co-op position  Possible part-time work  Interactive Collaboration Envroments Lab in

CASCI

 Collaboration Technology Specialists  Video / audio / communication over Internet 2.

 http://www.rit.edu/~rc/jobs.html

Jobs, Jobs, Jobs

 Upcoming company presentations:

 Google -- Tues, Feb 13, 4-6pm (70-1400)  Paychex -- Thurs, Feb 15, 2-4pm (in Webster -- RSVP

required)

 Intel -- Fri, Feb 16, noon-3pm (COE, Erdle)

 See posting on my office BB for info

Jobs, Jobs, Jobs

 US Air Force Research Labs

 Rome (New York, not Italy)  Summer Positions  Applications due February 28th

 Very hard deadline

 E-mail me for more info.

Projects

 Final Reports

 Note that final reports / code are

SEPARATE grading components

 Final reports/code are due on the last day

• f class (Feb 27th)

 Note this is a hard deadline  Dropbox is set up for report and code

submissions.

SLIDE 3

3 Projects

 Final Report

 Describe as if you were passing on your code to

the next victim.

 Sections

 Problem/Project Description  Approach  Implementation  Overall System Architecture.  Overall Program Architecture  Description of major data structures / objects  Results / User Documentation  Future Enhancements  Appendix -- All Code listings

Plan for today

 Sound and Animation

Motivation Films

 Animations by Wayne Lytle

 Visualization Guru at Cornell Theory Center  Quit to start Animusic in 1995

Motivational Film

 More Bells and Whistles (1990)

 Lytle wrote the the code for each band

member

 Motion is MIDI controlled  First of several Animusic pieces to be

shown at SIGGRAPH

Motivational Film

 Pipe Dream (2001)

 Animusic  Can’t See too much Animusic  Sound drives motion

Motivational Film

 Train Wreck (2003)  Martin Burolla  From animation class (20022)

SLIDE 4

4 Sound and Animation

 Issues in Sound and Animation

 Sound Generation

 What do we play?

 Sound Synchronization

 When do we play?

 Spatial Sound

 Where do we play

Sound

 What is sound?

 From webster.com

 mechanical radiant energy that is transmitted

by longitudinal pressure waves in a material medium (as air) and is the objective cause of hearing

Sound

 What is sound?

 Sound can be described as a 1 dimensional

signal in time sound = f(t)

Remember this?

 Spatial vs frequency domains

 Most well behaved functions can be

described as a sum of sin waves (possibly

• ffset) at various frequencies

 Describing a function by the contribution

(and offset) at each frequency is describing the function in the frequency domain

Sound

 A mathematical description of an audio

signal: ) 2 sin( ) (

i i i i

t A t f

• +

=

• =

Contribution/amplitude frequency phase

Sound

Foley/VanDam

SLIDE 5

5 Sound: Loudness

 Looking at sound in the temporal

domain

 Sound can be described as a 1 dimensional

signal in time

 Signal values represent amplitude.  We perceive the effect of amplitude as

loudness.

Sound: Pitch

 Looking at sound in the frequency

domain.

 Humans “hear” sounds because of

periodicities in the audio signal.

 Humans perceive frequency as the

sensation of pitch.

 Humans can perceive pitches due to

periodicities ranging from 20 – 20000 vibrations / sec (Hz).

Sound: Pitch

 Remember our discussion of CD audio

 sampling rate of 44,100 samples/sec  Δ = 1 sample every 2.26x10-5 seconds  CDs can accurately reproduce sounds with

frequencies as high as 22,050 Hz.

Sound: Timbre

 Tone quality of a sound  Formally defined as

 Characteristic of sound not due to

amplitude and pitch.

 Also defined

 Quality of tone that distinguishes between

musical instruments

 Sound shape

Sound: Timbre

 Timbre is the perception of the “spectral

makeup” of a signal.

 Adding non-fundamental frequency to the

signal.

 Another annoying audio applet – Timbre

Sound: Summary

Perceptual Characteristic Physical Characteristic Timbre Spectral “shape” Pitch Frequency Loudness Amplitude

SLIDE 6

6 Sound Generation

 So how does one generate sound for

animation?

 Easiest means

 Recording / Sampling -- Still the primary

means for sound generation in the film industry

 Using sampled sound – Still the primary means

for sound use in games.

Sound Generation

 When talking about digital (sampled

sound)

 The process of digitizing is called pulse

code modulation (PCM).

 PCM == sampled sound

 WAV  AIFF  MP3 (compressed PCM)

Sound Generation

 Additive Synthesis  Define values for Ai, ωi, and φi  Calculate sin and add  Alternately, do in the addition in frequency

space.

) 2 sin( ) (

i i i i

t A t f

• +

=

• =

Sound Generation

 Subtractive Synthesis  Start with noise (equal energies at all

frequencies)

 Subtract contribution of frequencies

from noise. ) 2 sin( ) (

i i i i

t A t f

• +

=

• =

Sound Generation

 Granular Synthesis

 Like particle system  Combine a multitude of sound “grains” into

a sound events

 Questions

Sound Generation

 Using physically based models

 SIGGRAPH 2001 Videos

SLIDE 7

7 Sound Synchronization

 Sound must be synchronized to the

motion

 Methods:

 Motion driving sound  Defining Sound events  Deriving timbre from motion  Sound driving motion

Sound Synchronization

 Sound driving motion

 MIDI

 Designed as a communication mode between

sythesizers, samplers, instruments, computers

 Sound events  Pitch  Devices  Used by Animusic in creating their videos

 Example video from Bingo guys.

Spatial Sound

 Sounds (and listeners) have spatial positions

 3D sound

 Making sounds appear as if they are emitted from a

given position accounting for listener position

 Reverberation

 Filtering of sound based on reflection off of environment

 Doppler Effect

 Change in pitch due to moving objects

3D Sound

 Making sounds appear as if they are

emitted from a given position accounting for listener position

 Head related transfer functions (HRTF)  Audio cubes / surround sound

 Strategic placing of speakers

3D Sound: HRTF

 a description of all the physical

cues of sound localisation.

 Implemented as filters  function of four variables: ie three

space coordinates and frequency.

 Determined by measurement

3D Sound: HRTF

Anderson/Casey

SLIDE 8

8 3D Sound: HRTF

Anderson/Casey

3D Sound: reverberation

 Like light, sound can be seen as

traveling in 3D environment in rays.

 Unlike light, sound travels much slower

 Speed of sound:  Speed of light

3D Sound: reverberation

 Reverberant sound is

the collection of all the reflected sounds in an enclosed space

 Acoustics  Reverb Time = time

required for sound to decay one millionth of the original power

3D Sound: reverberation

 Examples

 From BKL Consultants Ltd.

(http://www.bkl.ca/page131.htm)

 No reverb  0.8 sec reverb time  1.5 sec reverb time  5.0 sec reverb time

 One more SIGGRAPH video

3D Sound: Doppler effect 3D Sound: Doppler effect

 Non-annoying applet

break

SLIDE 9

9 Sound: Putting It all Together

Takala/Hahn

Sound: Putting It all Together

 Sound Rendering Video Examples

Approaches to sound generation

 Hueristic

 Whatever sounds good.

 Physically Based

 Analysis of vibration

 Emperical data

 Sampled Sound

Remember CGII: Procedural Shading

 Shade Trees [Cook84]

 Shading calculated by combining basic

functional operations.

 Operations are organized in a tree.

 Nodes - Operations  Children - operands

 Result of shade tree evaluation is a color  Equiv to parse tree (compiler design)  Basis of Renderman shading language.

Remember CGII: Procedural

Shading

 Shade Trees - example…copper [Cook84]

Remember CGII: Procedural

Shading

 Basic ideas behind shade trees:

 Describe textures / shading functionally  Using Parameters from 3D world

 Can we use a similar model for sound?

SLIDE 10

10 Timbre Trees

 Functional sound synthesis

 Sound related functions

 Periodic functions  Convolution  Noise  Filtering

 Nodes for animation, 3d parameters

Timbre trees

Hahn/Geigel. Et al

Timbre trees

 Nodes could also be used to simulate:

 Reverberation  Delay  Spatial Sound

Timbre trees

Hahn/Geigel. Et al

Timbre Trees

 What we failed to realize

 Functional sound, unlike functional

textures, was far from novel…

 Quite popular in the Computer Music circles

 Nyquist -- CMU  csound – MIT (basis of MPEG-4 Structured

Audio)

 However…

Genetic Texture

(sin (+ (- (grad-direction (blur (if (hsv- to-rgb (warped-color-noise #(0.57 0.73 0.92) (/ 1.85 (warped-color-noise x y 0.02 3.08)) 0.11 2.4)) #(0.54 0.73 0.59) #(1.06 0.82 0.06)) 3.1) 1.46 5.9) (hsv- to-rgb (warped-color-noise y (/ 4.5 (warped-color-noise y (/ x y) 2.4 2.4)) 0.02 2.4))) x))

[Sims91]

SLIDE 11

11 Genetic Sound

 Since Timbre trees were nothing more

than functional description of sound (using LISP expressions)

 Experimentation with genetic manipulation

was natural

Timbre Tree

 Video examples

Physically based sound generation

 Analysis of vibration

 SIGGRAPH 2001 Videos

 [O’Brien, Cook, Essl] – not real time  [van den Doel, Kry, Pai] – real time

Sampled sound

 The last…and most used approach

Good news about this research

 Sound now integrated as part of

rendering pipeline

 DirectSound  VRML2.0  openAL

Audio research

 Signal Processing  Multimedia (MM)  User Interface (CHI)  Computer Music (ICMC)  Data Sonification (ICAD)  Gaming (GDC)

SLIDE 12

12 Bad news about this research

 Sound effects for motion pictures is still

done using foley artists

Questions

 Next time

 Project presentations