Auditory Perception and Audition Addition of audio cues to VR - - PowerPoint PPT Presentation

CSE6335 Topics in Virtual Reality

Wednesday March 25, 2009

Auditory Perception and Audition

Addition of audio cues to VR environments Primary goal is to provide 3D audio cues (simulate audio coming from a particular direction) Technical problem is to do this with a limited array of speakers or a single set of headphones.

Sound in its simplest form

compression & rarefraction Wavelength/frequency Sound pressure level

Sound is a frequency-temporal process (can be represented as set of different frequencies at different times)

Threshold varies with frequency Ear as a filter

Middle Ear

Operates as an impedance transformer

Innear Ear

Organs of balance Cochlea

Pressure pathway through the Cochlea

Cochlea analyzes sound Auditory perception -> Audio in VR

Direction azimuth and elevation Distance Spaciousness Presence and envelopment

So how do we know where sounds come from?

Duplex Theory

We use ITD’s and IID’s to localize targets. Clearly not a complete theory (but predicts many effects). Cannot account for Long wavelength sounds (sound diffracts around the head). Geometric confusions..

Cone of confusion

Geometry shows a large number of points with the same ITD’s. So its more complex than just geometry

Effect of head rotation

Head rotations can be used to ‘solve’ front-back ambiguities

Head-related transfer function

Sound source localization is dependent upon the frequency distribution of sound sources. Each individual ear filters frequencies in different directions and we use this information to localize sounds. Note: makes assumptions about the distribution of energy in the sound source.

HRTFs

Can measure these individually, construct ‘approximate’ ones, etc. KEMAR is a freely available one. HRTF Measurement

So,

This suggests how to simulate 3d sound in VR Generate appropriate IID and ITD cues Generate appropriate HRTF cues Present these cues to the left and right ears.

Headphone systems

One ‘speaker’ to each ear. Invalidates the effects

• f the pinnae.

Suffers from ‘inside the head’ localization.

Speaker systems

2, 4, 5.1 and higher channel systems exist Question is ‘how do you spread sound over these speakers so that the sound comes from a specific direction?’ The first big problem is crosstalk

Crosstalk

How do you ‘cancel out’ the unwanted sound?

Reducing crosstalk

Generate inverted (negative) version of appropriately delayed sound to ‘cancel’ out as much as possible the unwanted sound. Can do this recursively.

Distributing sound over a speaker array

Basic task is to apply amplitude gains (and possibly temporal delays) to sound given the sound location and distribution of speakers. Many different approaches, we will look at two

Ambisonics

Amplitude-panning approach Sound has a gain factor at each microphone

alpha is the angular separation between source and speaker i Four channel system

Vector base amplitude panning Triplet-based system

Direction to speaker n Direction to sound source p is a linear sum

Audition and VE

VE relies on portaudio to actually talk to k speakers You write code that takes m input audio channel and distributes it over the k output channels. Your code is executed in real time as needed

Audition and VE

Simple left-right % mixer shown

Summary

Goal is to simulate 3d sound ITD, IID and HRTF cues Headphone and speaker array systems Basic approach is to distribute acoustic cues over the available sound sources