Continuous gesture control of audio and visual media Nicolas - - PowerPoint PPT Presentation

▶

Feb 17, 2024 159 likes •293 views

Continuous gesture control of audio and visual media Nicolas Rasamimanana, Bruno Zamborlin, Frdric Bevilacqua, Norbert Schnell, Fabrice Gudy IRCAM, CNRS - STMS, Real Time Musical Interactions Team 1 Place Igor Stravinsky, 75004 Paris,

SLIDE 1

Continuous gesture control of audio and visual media

Nicolas Rasamimanana, Bruno Zamborlin, Frédéric Bevilacqua, Norbert Schnell, Fabrice Guédy

IRCAM, CNRS - STMS, Real Time Musical Interactions Team 1 Place Igor Stravinsky, 75004 Paris, France Frederic.Bevilacqua@ircam.fr

SLIDE 2

IRCAM - Real Time Musical Interactions

Motivations - Context

Expressive gestural control of digital media

Embodiment in music Musical Expression (NIME)

Applications in music and performing arts

Artistic and pedagogical applications New interfaces (cell phone, Wii, etc)

from « triggering » to « following »

push the button lead the electronic music

SLIDE 3

IRCAM - Real Time Musical Interactions

Goals

Hyp: Gesture « meaning » is in temporal evolutions
Real-time gesture analysis :

gesture following: time progression of the

performed gesture

recognition/characterization: similarity of the

performed gesture to prerecorded gestures

Requirements

simple learning procedure, with a single example adaptation to the user idiosyncrasies continuous analysis from the beginning of the gestures

SLIDE 4

IRCAM - Real Time Musical Interactions

Demo max

SLIDE 5

IRCAM - Real Time Musical Interactions

Time Profile Modeling: HMM

time sensor value

probability density function

Markov Chains

transition probabilities

Markov Models

SLIDE 6

IRCAM - Real Time Musical Interactions

Gesture ?

Any datastream of continuous parameters
typically 0.1 to 1000 Hz
from motion capture systems:

image descriptors accelerometers, gyroscope, magnetometers

from sound descriptors

pitch, loudness mfccs, ...

multimodal data

SLIDE 7

IRCAM - Real Time Musical Interactions

Music Pedagogy Applications

Continuous control of audio

SLIDE 8

IRCAM - Real Time Musical Interactions

Music Pedagogy Applications

SLIDE 9

IRCAM - Real Time Musical Interactions

Artistic Applications

augmented string quartet (StreicherKreis - Florence Baschet)

SLIDE 10

IRCAM - Real Time Musical Interactions

Artistic Applications

Dance performance - Continuous control of video

SLIDE 11

IRCAM - Real Time Musical Interactions

Implementation

gf: C++ libs iPad mobile device Max/MSP

+ FTMCo and Mubu Library : real time control of multimodal data and sound synthesis.

SLIDE 12

IRCAM - Real Time Musical Interactions

Conclusions

Reliable system for realtime
continuous time warping (time progression)
time profile characterization based on comparison with

examples, recognition possible

Successfully implemented in artistic and pedagogical

applications in performing arts

Current work:
anticipation, prediction
adaptive system
Demos at Barcamp

SLIDE 13

IRCAM - Real Time Musical Interactions

Acknowledgements

We acknowledge partial support of the following projects:
ANR project Interlude (France).
EU-ICT Project SAME (Sound And Mu- sic for Everyone Everyday

Everywhere Every way, http://www.sameproject.eu/)

i-Maestro project (IST
026883, www.i-maestro.org),
ANR project EarToy (France).
Thanks to

Atelier les Feuillantines : Fabrice Guédy, Barnabé, Sandro, Berenice Remy Muller, Alice Daquet, Riccardo Borghesi, Diemo Schwarz,

Fivos Maniatakos, Tommaso Bianco, Donald Glowinski