Active Audition and Sensorimotor Integration for Sound Source - - PowerPoint PPT Presentation

Active Audition and Sensorimotor Integration for Sound Source Localization

Mathieu Bernard 25 novembre 2011

1/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Introduction

CIFRE thesis. Co-direction : Patrick Pirim, Brain Vision Systems Bruno Gas, ISIR, UPMC Alain de Cheveign´ e, LPP, Paris Descartes Outline Artificial auditory system for sound localization, Audio-tactile model for texture recognition, Active audition and sensorimotor integration.

2/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Autonomous robotics

Psikharpax, the robot rat

3/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Artificial auditory system

Bioinspired sound localization

Binaural localiation cues ITD - ILD Outer ear Outer ear Inner ear Inner ear

Auditory model

Implementation

core library real time - c++ robot robotic simulation standalone programs

sound capture simulated sound sources wav files

4/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Outer ear model

Pinna Auditory fovea Microphone Oriented toward the fovea Support Servomotor

Outer ear = Pinna, microphone and software capture. Spectral and directional cues.

5/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Inner ear model (1/2)

Inner ear = cochlear model and pulse train generation. Cochlear model Gammatone filterbank Usually 30 channels from 300Hz to 8kHz at 20kHz

6/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Inner ear model (2/2)

Inner ear = cochlear model and pulse train generation. Pulse train generation Cochlear channel output ⇒ Pulse train Discrete representation, noise suppression. A pulse = temporal and amplitude information.

7/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

ILD computation (1/2)

Energy for each pulse p(t), the energy is : E(t) =

u=t

• u=t−T

p(u)2 (1)

8/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

ILD computation (2/2)

ILD = Interaural Level Difference For each channel i : ILDi(t) =

2Eleft,i(t) Eleft,i(t)+Eright,i(t) − 1

9/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

ITD computation (1/3)

ITD = Onset extraction and delay lines. Onset extraction from a pulse train Comparison with a dynamic threashold, 2 parameters.

10/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

ITD computation (2/3)

ITD = Onset extraction and delay lines. Delay lines model

11/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

ITD computation (3/3)

ITD = Onset extraction and delay lines.

12/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Auditory and tactile model for texture perception (1/3)

Similar model for transduction and processing Audition and touch support fine texture discrimination skills. Rat vibrissae and cochlea transduction based on resonance. Strong interaction for auditory/tactile spectral processing.

13/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Auditory and tactile model for texture perception (2/3)

Whiskers filterbank adapted from gammatone cochlear model : Feature extraction = Instantaneous Mean Power :

14/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Auditory and tactile model for texture perception (3/3)

Texture classification : 8 textures, 3 layer perceptron Response to a pure tone (whisker at 630 Hz) Influence of the number of channels

15/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Auditory Evoked Behaviors (1/2)

ILD Outer ear Inner ear Energy Outer ear Inner ear

Auditory model Motor control

Neck and body Energy

Motor control Neck ⇒ ILD minimization ILD < 0 ⇒ turn right, ILD > 0 ⇒ turn left. Video ! Wheels ⇒ Follow neck orientation Constant speed, Smooth rotations. Video !

16/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Auditory Evoked Behaviors (2/2)

Phonotaxis trajectories. Front-back disambiguation.

17/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Sensorimotor Approach (1/3)

Environment state e ∈ E and motor state m ∈ M, Sensory state s ∈ S, we have s = Φ(m, e), Φ is called a sensorimotor law.

18/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Sensorimotor Approach (2/3)

Quand on dit [...] que nous localisons tel objet en tel point de l’espace [...] cela signifie simplement que nous nous repr´ esentons les mouvements qu’il faut faire pour atteindre cet objet. [...] Nous nous repr´ esentons les sensations musculaires qui accompagnent [ces mouvements] et qui n’ont aucun caract` ere g´ eom´ etrique, qui par cons´ equent n’impliquent nullement la pr´ eexistance de la notion d’espace.

• H. Poincar´

e, L’espace et la g´ eom´ etrie, 1845. Proposed formalization for localization Find the motor state ˜ m such as ˜ m = argmin

m∈M

|Φ(m, e) − Φ(m0, e0)|.

19/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Sensorimotor Approach (3/3)

Sound source localization Two ways for ˜ m estimation Orienting behavior After completion we have ˜ m = mend send = Φ(mend, e) = Φ(m0 + δm, e0 + δe), Manifold learning Sensory space S lies on a low-dim manifold R, Dimension reduction technique : S → R, Same topology as the embodying space (at least locally).

20/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Manifold learning (1/3)

CAMIL database, ILD vectors :

21/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Manifold learning (2/3)

CAMIL database, ITD vectors : d=690 → d=3

22/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Manifold learning (3/3)

2 outer ears models : HRTF and directive filters Front-back disambiguation with spectral cues :

23/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Sensorimotor integration (1/2)

Learning algorithm Iterative learning of R, Self-supervised.

24/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Sensorimotor integration (2/2)

Simulation results over 400 experiments

25/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Intrinsic Dimension Estimation

Simulation (ILD, azimtuh 180 & 360) CAMIL dataset (ILD & ITD, azimtuh 360, elevation [-30, 30])

26/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

Perspectives

SensoriMOTOR

27/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration