REAL-TIME SYSTEMS FOR GESTURE-SOUND INTERACTION Jules Franoise - - PowerPoint PPT Presentation

▶
real time systems for gesture sound interaction
SMART_READER_LITE
LIVE PREVIEW

REAL-TIME SYSTEMS FOR GESTURE-SOUND INTERACTION Jules Franoise - - PowerPoint PPT Presentation

Mar 28, 2023 223 likes •506 views

REAL-TIME SYSTEMS FOR GESTURE-SOUND INTERACTION Jules Franoise Real-Time Musical Interactions Team (IMTR) IRCAM jules.francoise@ircam.fr http://imtr.ircam.fr Interactive musical system ? !"#$"%&'() *$&'() 2 Interactive

slide-1
SLIDE 1

REAL-TIME SYSTEMS FOR GESTURE-SOUND INTERACTION

Jules Françoise Real-Time Musical Interactions Team (IMTR) IRCAM

jules.francoise@ircam.fr http://imtr.ircam.fr

slide-2
SLIDE 2

Interactive musical system ?

2

!"#$"%&'() *$&'()

slide-3
SLIDE 3

Interactive musical system ?

2

research: gesture & sound

Human-Computer Interaction (HCI) Music, Performing arts Perception, Cognitive sciences Signal processing, Machine learning

!"#$"%&'()

!"#$%&" '()$%&"*+*,-(./#0# 1())0-2 34%-5*3/-$6"#0#

*$&'()

slide-4
SLIDE 4
  • 1. Gesture-sound interaction paradigms
  • 2. Formalizing Gesture-sound relationships

Overview

slide-5
SLIDE 5

GESTURE-SOUND INTERACTION PARADIGMS

slide-6
SLIDE 6

ANR Project Interlude (2008-2011)

design: J-L Frechin and U. Petrevski

Ircam - Grame - Da Fact - Voxler - Atelier des Feuillantines - nodesign.net

New interfaces for musical expression

5

slide-7
SLIDE 7

Continuous Control

6

sensors (audio) Audio analysis Granular synthesis

slide-8
SLIDE 8

Triggering

7

sensors kick detection sample triggering segmented audio recording

slide-9
SLIDE 9

Sound Selection

8

audio database audio samples classified by loudness sensors gesture analysis (movement energy) energy comparison sound selection external clock

slide-10
SLIDE 10

Navigation: CataRT

9

Diemo Schwarz, 2005-2011

sound selection cursor (x,y) external clock audio stream segmentation descriptor extraction audio database audio units distributed in a descriptor space sensors gesture analysis

slide-11
SLIDE 11

Navigation

10

slide-12
SLIDE 12

Intermediate Physical Model

11

sensors Parameter extraction (inclination) sound selection audio database audio samples classified by descriptore parameter comparison Physical Model synchronization

slide-13
SLIDE 13

Machine learning: Gesture Follower

12

slide-14
SLIDE 14
  • Common gesture recognition systems:
  • Gesture = unit → recognition after completion
  • Simple iconic gestures
  • Music and performing arts:
  • Which gesture ?
  • guess from the gesture beginning ?
  • How is the gesture performed ?
  • Motion as a continuous process:
  • monitor the gesture progression ?

→ Development of the “gesture follower”

Gesture and action recognition

13

slide-15
SLIDE 15

time gesture parameter

recorded example performed gesture (live)

  • Synchronization/following
  • Recognition
  • Anticipation (prediction)

14

Real-time time warping

slide-16
SLIDE 16

15

Aligning dance sequence on prerecorded videos

Richard Siegal, Jean-Philippe Lambert, using the gesture follower Ircam

slide-17
SLIDE 17

Gesture Follower: synchronization

16

performed gesture model: encodes temporal structure of the gesture Alignment reference gesture Audio / video recording time streching Learning { Performance{

slide-18
SLIDE 18
slide-19
SLIDE 19

Première in 2008 with Quatuor Danel Computer Music Design: Serge Lemouton Commissioned by IRCAM-Centre Pompidou, 2008

for augmented string quartet

“gesture” =

  • acceleration
  • angular velocity
  • pressure
  • audio energy

StreicherKreis - Florence Baschet

18

slide-20
SLIDE 20

vl 1 vl 2 viola cello

acceleration

Gesture Follower - Time Warping

19

slide-21
SLIDE 21

FORMALIZING GESTURE-SOUND RELATIONSHIPS

slide-22
SLIDE 22

Overview of the paradigms

21

Instantaneous Mapping Temporal Mapping

Machine learning

slide-23
SLIDE 23

Overview of the components

22

audio database sensors gesture analysis sound selection external clock machine learning (gesture modeling) sound synthesis sample triggering time stretching dimensionality reduction machine learning (sound modeling) audio segmentation audio descriptor extraction

?

....

slide-24
SLIDE 24

General formalization of these relationships ? Unified framework for gesture-sound interactions Key issues:

  • Multimodality (≠ resolutions)
  • Combining different paradigms
  • Multiple temporal scales
  • Adding hierarchy
  • Dealing with multiple interacting systems / performers
  • Synchronization
  • group interaction

Formalization

23

slide-25
SLIDE 25

Can we learn gesture-sound relationships from example ?

  • Current system: Gesture Follower

to be extended (geometric invariants, hierarchy)

  • Use learning for other paradigms
  • Learn in complex situations

superposition of paradigms different temporal scales

Learning gesture-sound couplings

24

slide-26
SLIDE 26

Credits and Acknowledgements

25

Real-Time Musical Interactions team: http://imtr.ircam.fr

Frédéric Bevilacqua, Tommaso Bianco, Julien Bloit, Ricardo Borghesi, Baptiste Caramiaux, Arshia Cont, Arnaud Dessein, Sarah Fdili Alaoui, Emmanuel Fléty, Vassilios-Fivos Maniatakos, Norbert Schnell, Diemo Schwarz, Fabrice Guédy, Alain Bonardi, Nicolas Rasamimanana, Bruno Zamborlin, Jean-Philippe Lambert

With support of the projects:

ANR projects: Interlude, Topophonie (France) EU-ICT project SAME Inside Movement Knowledge Thanks to Richard Siegal/The Bakery