RESound Interactive Sound Rendering in Dynamic Virtual Environments - - PowerPoint PPT Presentation

▶
resound
SMART_READER_LITE
LIVE PREVIEW

RESound Interactive Sound Rendering in Dynamic Virtual Environments - - PowerPoint PPT Presentation

Dec 25, 2022 205 likes •962 views

RESound Interactive Sound Rendering in Dynamic Virtual Environments Micah Taylor, Anish Chandak Lakulish Antani, Dinesh Manocha University of North Carolina ACM MM 09 1 October 19-23, 2009 Sound rendering and applications Details of

slide-1
SLIDE 1

ACM MM 09 October 19-23, 2009

1

RESound

Interactive Sound Rendering in Dynamic Virtual Environments

Micah Taylor, Anish Chandak Lakulish Antani, Dinesh Manocha

University of North Carolina

slide-2
SLIDE 2

ACM MM 09 October 19-23, 2009

2

  • Sound rendering and applications
  • Details of propagation
  • Our system: RESound
slide-3
SLIDE 3

ACM MM 09 October 19-23, 2009

3

  • Sound rendering and applications
  • Details of propagation
  • Our system: RESound
slide-4
SLIDE 4

ACM MM 09 October 19-23, 2009

4

Sound Rendering

  • Three main steps
  • Signal input
  • Sound propagation
  • Audio output
slide-5
SLIDE 5

ACM MM 09 October 19-23, 2009

5

Sound Rendering: Signal Input

  • Recorded sample
  • Simple and fast
  • Played with events
  • Static
  • Synthesized sound
  • Physics simulation

generates sound

  • Matches virtual events

[Raghuvanshi 2006] [Matt Hileo]

slide-6
SLIDE 6

ACM MM 09 October 19-23, 2009

6

Sound Rendering: Signal Input

  • Synthesized sound
  • Uses physical models

[Florens et al. 1991]

  • Interactive rates with

many objects

[Raghuvanshi et al. 2006]

  • Correlates closely with

visual scene

[Ren et al. 2009]

slide-7
SLIDE 7

ACM MM 09 October 19-23, 2009

7

Sound Rendering: Propagation

  • Goal: Model environment influences
  • Echoes
  • Delay from distance
  • Attenuation from distance
  • Frequency shifts
  • Output: Impulse response
  • Represents room's effect on input signal
slide-8
SLIDE 8

ACM MM 09 October 19-23, 2009

8

Sound Rendering: Propagation

  • Common methods
  • No propagation - direct path only
  • Geometric simulation
  • Numerical simulation
slide-9
SLIDE 9

ACM MM 09 October 19-23, 2009

9

Sound Rendering: Audio Output

  • Goal
  • Combine many sounds from environment
  • Apply any needed effects
  • Output to user's audio device
  • Uses the output from prior steps
  • Input signal
  • Room impulse response
slide-10
SLIDE 10

ACM MM 09 October 19-23, 2009

10

Sound Rendering: Audio Output

  • Common output methods
  • Mono

– Fast, simple – No spatialization

  • Stereo

– Fast, simple, left+right spatialization

  • 3d sound

– Head Related Transfer Functions (HRTF) – Complex, very good spatialization

slide-11
SLIDE 11

ACM MM 09 October 19-23, 2009

11

Applications

  • Video games
  • Helps player avoid

monsters

  • Provides sound cues

to environment size

  • Used in most 3d video

games

slide-12
SLIDE 12

ACM MM 09 October 19-23, 2009

12

Applications

  • Training simulators
  • Improves realism
  • Decreases incorrect

training

  • Current uses
  • Tactical training
  • EMT training
slide-13
SLIDE 13

ACM MM 09 October 19-23, 2009

13

Applications

  • Multimedia
  • Auditory displays

– Enhance data

visualization

  • Telephony and Video

conferencing

slide-14
SLIDE 14

ACM MM 09 October 19-23, 2009

14

Applications

  • Computer aided

design

  • Relay cues about

environment design

  • Preview room

acoustics before construction

slide-15
SLIDE 15

ACM MM 09 October 19-23, 2009

15

  • Sound rendering and applications
  • Details of propagation
  • Our system: RESound
slide-16
SLIDE 16

ACM MM 09 October 19-23, 2009

16

Propagation

  • Simplest method:
  • Direct path between

source and listener

  • Add echoes with post-

process filter

  • Fast
  • Widely used
slide-17
SLIDE 17

ACM MM 09 October 19-23, 2009

17

Propagation

  • However
  • Not physically based
  • Spatialization

incorrect

  • Echoes do not match

environment

slide-18
SLIDE 18

ACM MM 09 October 19-23, 2009

18

Propagation

  • Acoustic simulations
  • Numerical

– Solves acoustic wave equation – Slow, but getting faster [Raghuvanshi et al. 2009]

  • Geometric

– High frequency approximation – Very fast – interactive – Models sound as ray

slide-19
SLIDE 19

ACM MM 09 October 19-23, 2009

19

Propagation

  • Specular reflection
  • Mirror-like reflections
  • Reflections decrease

amplitude

  • Longer paths,

longer delays

slide-20
SLIDE 20

ACM MM 09 October 19-23, 2009

20

Propagation

  • Specular reflection
  • Mirror-like reflections
  • Reflections decrease

amplitude

  • Longer paths,

longer delays

  • Often many reflection

paths

slide-21
SLIDE 21

ACM MM 09 October 19-23, 2009

21

Propagation

  • Diffuse reflection
  • Scattering reflections
slide-22
SLIDE 22

ACM MM 09 October 19-23, 2009

22

Propagation

  • Diffuse reflection
  • Scattering reflections
  • Scattered sound

reaches listener

slide-23
SLIDE 23

ACM MM 09 October 19-23, 2009

23

Propagation

  • Diffraction
  • Sound 'bends' around

corners

  • Can change phase
slide-24
SLIDE 24

ACM MM 09 October 19-23, 2009

24

Propagation

  • Diffraction
  • Sound 'bends' around

corners

  • Can change phase
  • Often many diffraction

paths

slide-25
SLIDE 25

ACM MM 09 October 19-23, 2009

25

Propagation

  • Combine
  • Direct
  • Specular
  • Diffuse
  • Diffraction
  • Early contributions
  • 4-5 recursions
slide-26
SLIDE 26

ACM MM 09 October 19-23, 2009

26

Propagation

  • Reverberation
  • Late contributions
  • Impulses decays over

time

  • Hundreds of

recursions

  • Gives 'feel' of the

room

slide-27
SLIDE 27

ACM MM 09 October 19-23, 2009

27

Propagation

  • Specular reflections
  • Image-source method [Allen et al. 1979]
  • From source
  • Reflect against all scene triangles

– Creates image-sources – Is listener visible

  • Reflect image sources

– and so on...

slide-28
SLIDE 28

ACM MM 09 October 19-23, 2009

28

Propagation

  • However
  • Very compute intensive
  • Need to accelerate
  • Graphic acceleration
  • Remove non-visible triangles
  • Sound acceleration
  • Remove non-reflecting triangles
slide-29
SLIDE 29

ACM MM 09 October 19-23, 2009

29

Propagation

  • Accelerated by
  • Ray tracing [Vorlander 1989]
  • Beam tracing [Funkhouser et al. 1998]
  • Frustum tracing [Lauterbach et al. 2007]
  • And others...
  • Often require precomputation
  • Non-moving source
slide-30
SLIDE 30

ACM MM 09 October 19-23, 2009

30

Propagation

  • Diffuse reflections
  • Often modeled by ray tracing [Dalenbaeck 1996]
  • Radiosity [Siltanen et al. 2004]
  • Compute intensive
  • Fixed source and receiver
  • No scene movement
slide-31
SLIDE 31

ACM MM 09 October 19-23, 2009

31

Propagation

  • Diffraction
  • Added to

– Beam tracing [Tsingos et al. 2001] – Ray tracing [Stephenson et al. 2007] – Frustum tracing [Taylor et al. 2009] – Image source [Shroeder et al. 2009]

slide-32
SLIDE 32

ACM MM 09 October 19-23, 2009

32

Propagation

  • Reverberation
  • Ray tracing

– Slow, accurate [Hodgson 1990]

  • Statistical

– Fast, some error [Savioja et al. 1999]

slide-33
SLIDE 33

ACM MM 09 October 19-23, 2009

33

  • Sound rendering and applications
  • Details of propagation
  • Our system: RESound
slide-34
SLIDE 34

ACM MM 09 October 19-23, 2009

34

RESound

  • Simulates all mentioned effects
  • Interactive update rates
  • Dynamic scenes
  • Handles propagation and output
  • Given input sound + environment
  • Propagates sound through environment
  • Renders signal at receiver's position
slide-35
SLIDE 35

ACM MM 09 October 19-23, 2009

35

RESound

System overview

slide-36
SLIDE 36

ACM MM 09 October 19-23, 2009

36

RESound

  • Early contributions by simulation
  • Specular + diffraction
  • Diffuse reflection
  • Late contributions by statistics
  • 3d audio output
slide-37
SLIDE 37

ACM MM 09 October 19-23, 2009

37

RESound

  • Unified engine
  • Frustum tracing
  • Ray tracing
  • Ray primitive
  • Single acceleration structure
  • Bounding Volume Hierarchy
  • Allows dynamic scenes
  • Fast ray tracing
slide-38
SLIDE 38

ACM MM 09 October 19-23, 2009

38

RESound

  • Scene acceleration hierarchy
  • Bounding Volume Hierarchy [Lauterbach et al. 2006]

– Fast construction times – Allows interactive visual ray tracing – Allows dynamic scene changes

  • Can accelerate frustum and ray tracing
slide-39
SLIDE 39

ACM MM 09 October 19-23, 2009

39

  • Specular + diffraction
  • Frustum tracing
  • Volumetric, finds most

paths

  • Dynamic scenes
  • Fast
  • Diffuse
  • Ray tracing
  • Shares scene

structure

  • Dynamic scenes
  • Fast
slide-40
SLIDE 40

ACM MM 09 October 19-23, 2009

40

RESound

  • Frustum tracing
  • Specular reflection
slide-41
SLIDE 41

ACM MM 09 October 19-23, 2009

41

RESound

  • Frustum tracing
  • Specular reflection
  • Frustum is bounded

by rays

slide-42
SLIDE 42

ACM MM 09 October 19-23, 2009

42

RESound

  • Frustum tracing
  • Specular reflection
  • Check if receiver is

inside bounded volume

slide-43
SLIDE 43

ACM MM 09 October 19-23, 2009

43

RESound

  • Frustum tracing
  • Specular reflection
  • Bounding rays can be

reflected

slide-44
SLIDE 44

ACM MM 09 October 19-23, 2009

44

RESound

  • Frustum tracing
  • Specular reflection
  • Sound path is linear

combination of rays

slide-45
SLIDE 45

ACM MM 09 October 19-23, 2009

45

RESound

  • Diffraction
  • Covers more area
  • Allows smooth

transitions

– Fades out

slide-46
SLIDE 46

ACM MM 09 October 19-23, 2009

46

RESound

  • Diffraction
  • Covers more area
  • Allows smooth

transitions

– Fades out

  • First step
  • Find diffracting edges
slide-47
SLIDE 47

ACM MM 09 October 19-23, 2009

47

RESound

  • Frustum tracing
  • Edge diffraction
slide-48
SLIDE 48

ACM MM 09 October 19-23, 2009

48

RESound

  • Frustum tracing
  • Edge diffraction
  • From source

– Trace many frusta

slide-49
SLIDE 49

ACM MM 09 October 19-23, 2009

49

RESound

  • Frustum tracing
  • Edge diffraction
  • Receiver is

hidden from source

slide-50
SLIDE 50

ACM MM 09 October 19-23, 2009

50

RESound

  • Frustum tracing
  • Edge diffraction
  • But diffracting edge

is visible

slide-51
SLIDE 51

ACM MM 09 October 19-23, 2009

51

RESound

  • Frustum tracing
  • Edge diffraction
  • Create diffraction

frustum

slide-52
SLIDE 52

ACM MM 09 October 19-23, 2009

52

RESound

  • Frustum tracing
  • Edge diffraction
  • Diffracting sound

reaches the receiver

slide-53
SLIDE 53

ACM MM 09 October 19-23, 2009

53

RESound

  • Diffuse reflections
  • Uses ray tracing
  • Collection sphere
  • Same size as listener's head (0.3 m)
  • Record rays that hit collection sphere
slide-54
SLIDE 54

ACM MM 09 October 19-23, 2009

54

RESound

  • Ray tracing
  • Diffuse reflection
slide-55
SLIDE 55

ACM MM 09 October 19-23, 2009

55

RESound

  • Ray tracing
  • Diffuse reflection
  • Shoot rays from

source

slide-56
SLIDE 56

ACM MM 09 October 19-23, 2009

56

RESound

  • Ray tracing
  • Diffuse reflection
  • Rays diffusely reflect
slide-57
SLIDE 57

ACM MM 09 October 19-23, 2009

57

RESound

  • Ray tracing
  • Diffuse reflection
  • Some rays hit this

collection sphere

slide-58
SLIDE 58

ACM MM 09 October 19-23, 2009

58

RESound

  • Update stronger paths more often:
  • Three simulations
  • Frustum tracing (first order, 1 thread)
  • Frustum tracing (third order, 7 threads)
  • Ray tracing, 200k rays (third order, 7 threads)
slide-59
SLIDE 59

ACM MM 09 October 19-23, 2009

59

RESound

  • From 3 simulations
  • Now have impulse response of:
  • Direct sound
  • Specular reflection
  • Diffuse reflection
  • Edge diffraction
slide-60
SLIDE 60

ACM MM 09 October 19-23, 2009

60

RESound

  • Audio output
  • Reverberation
  • 3d sound rendering
  • Dynamic scenes
slide-61
SLIDE 61

ACM MM 09 October 19-23, 2009

61

RESound

  • Reverberation
  • Need to fill in late contributions
  • Use Eyring model [Eyring 1930]
  • Statistically estimate sound decay
  • Combing impulse responses
  • Frustum + frustum + ray tracing
slide-62
SLIDE 62

ACM MM 09 October 19-23, 2009

62

RESound

  • Reverberation
  • Fit curve to impulse response
  • Estimate time for signal to decay to 0.001% (RT60)
  • Create reverberation filter with sound system
slide-63
SLIDE 63

ACM MM 09 October 19-23, 2009

63

RESound

  • HRTF is expensive
  • Three impulse responses

– 1st order frustum tracing – 3rd order frustum tracing – 3rd order ray tracing

  • Compute only for 1st order frustum tracing
  • Other impulses use simple convolution
slide-64
SLIDE 64

ACM MM 09 October 19-23, 2009

64

RESound

  • Dynamic scenes
  • Impulse response may change drastically
  • Can cause artifacts (clicking)
  • Restrict motion speed
  • Crossfade audio frames
slide-65
SLIDE 65

ACM MM 09 October 19-23, 2009

65

Results

  • Test scenes

Room Conference Sibenik Sponza

slide-66
SLIDE 66

ACM MM 09 October 19-23, 2009

66

Results

  • Open scenes
  • Many triangles visible
  • Many reflections
slide-67
SLIDE 67

ACM MM 09 October 19-23, 2009

67

Results

slide-68
SLIDE 68

ACM MM 09 October 19-23, 2009

68

Results

  • Reverberation
  • Begin with 6m cathedral
  • Dynamically expand cathedral to 30m
  • With reverb and without
slide-69
SLIDE 69

ACM MM 09 October 19-23, 2009

69

Results

slide-70
SLIDE 70

ACM MM 09 October 19-23, 2009

70

Results

  • Limitations
  • Must shoot many rays for diffuse reflections
  • Certain diffraction paths may not be found
  • Frustum tracing is approximate visibility

– May miss some paths

slide-71
SLIDE 71

ACM MM 09 October 19-23, 2009

71

Results

  • Specular + diffuse + diffraction components
  • Uses unified representation: ray
  • Single acceleration structure
  • Interactive rates on multi-core PC
  • Handles large scenes
  • Moving source and listener
  • Scene can be dynamic
slide-72
SLIDE 72

ACM MM 09 October 19-23, 2009

72

Related and Future Work

  • Conservative frustum tracing [Chandak et al. 2009]
  • GPU acceleration
  • Robust diffraction
  • Conservative diffraction region
  • From region visibility – advanced diffraction
slide-73
SLIDE 73

ACM MM 09 October 19-23, 2009

73

Acknowledgements

  • Nikunj Raghuvanshi and Paul Calamia for

helpful advice

  • Sponsors

– ARO – NSF – DARPA/RDECOM – Intel – Microsoft

slide-74
SLIDE 74

ACM MM 09 October 19-23, 2009

74

Project website http://gamma.cs.unc.edu/Sound/RESound/

Thanks!