MuBu IMTR IRCAM Centre Pompidou Norbert Schnell Riccardo Borghesi - - PowerPoint PPT Presentation

▶
mubu
SMART_READER_LITE
LIVE PREVIEW

MuBu IMTR IRCAM Centre Pompidou Norbert Schnell Riccardo Borghesi - - PowerPoint PPT Presentation

Jul 05, 2023 272 likes •526 views

MuBu IMTR IRCAM Centre Pompidou Norbert Schnell Riccardo Borghesi 20/10/2010 Motivation Create a solid and open framework for the experimentation with recorded data streams of multiple representations in Max/MSP - audio samples - audio

slide-1
SLIDE 1

MuBu

IMTR IRCAM – Centre Pompidou Norbert Schnell Riccardo Borghesi 20/10/2010

slide-2
SLIDE 2

Motivation

Create a solid and open framework for the experimentation with recorded data streams of multiple representations in Max/MSP

  • audio samples
  • audio descriptors
  • gesture and motion capture data
  • spectral audio representations
  • symbolic representations
  • segmentation and annotations

... what else?

slide-3
SLIDE 3

Converging Applications

  • Analysis/re-synthesis (sound, music and speech)
  • Corpus based granular synthesis (textures)
  • Following and recognition (gestures/sounds and scores)
  • Computer aided improvisation
  • ...
slide-4
SLIDE 4

Context

  • Créations IRCAM & Forum
  • Recherche musicale
  • Prototyping of interactive real-time applications
  • Application development (plug-ins, iPhone, etc.)
slide-5
SLIDE 5

Relaxed Real-Time Analysis/Synthesis

  • Real-time analysis, recognition and classification
  • structured sound and gesture data
  • “emerging symbolics”
  • Content based audio processing and rendering
  • synthesis based on offline or real-time analysis
  • “post-sampling” and “re-performance”
slide-6
SLIDE 6

MuBu Container Functionalities

  • Store heterogeneous synchronous data streams
  • File import/export
  • Visualization and editing
  • Optimized congruent access to shared data
  • Handling of dynamic configurations (data and connections)
slide-7
SLIDE 7

MuBu

Container real-time/offline analysis real-time synthesis

A S

file import

F G

graphical editing

Data Models

slide-8
SLIDE 8

MuBu Container Design

  • Simple and easy to maintain
  • Minimizing dependencies on other libraries
  • Thread-safe and lock-free congruent data access
slide-9
SLIDE 9

Container Data Structure

  • Array of buffers being arrays of tracks

track 1

...

track 2 track 3 track T

...

buffer 1 buffer 2 buffer 3 buffer B

slide-10
SLIDE 10

Track Data Structure

  • Array of 2-dim matrix data (optional time-tags and matrix rows)
  • Array of (non-matrix) “extra” data (currently labels only)
  • Track meta-data

parameters info dictionary matrix column names

...

time-tags (64-bit float)

... ... ...

variable number of matrix rows (int) extra data (symbols or other) N x M matrix data (32-bit float)

slide-11
SLIDE 11

Modularity and Thread-Safeness

  • Distinction of configuration and real-time access
  • thread-locked access to container (using pthread r/w locks)
  • lock-free access to track data (using compare-and-swap)
  • Track implemented as immutable data structure

→ fixed capacity → re-configuration by copying data into a new track

  • Container manages dynamic track configuration
  • locked callbacks to modules that refer to a container
  • simple garbage collector (by reference count)
slide-12
SLIDE 12

File Format Support

  • SDIF import and export*
  • Plain text file import and export
  • MIDI standard file import
  • MusicXML import
  • Save/load data with Max patcher (for small amounts of data)

* ... what else?

* features to be implemented

slide-13
SLIDE 13

Visualization and Editing

  • Based on the Juce framework
  • Abstract data interface (also implemented for FTM data structures)
  • Set of editor/visualization components
  • waveform (single or multi-channel)
  • break-point function (single or also multi-channel)
  • sonogram (of sampled or time-tagged data)
  • markers (with duration and offset)
  • textual tables/matrices
  • piano roll (or simplified staffs)
  • traces (waveform or bpf with color and thickness)
  • Control components (scroll bar, rulers, toolbar, tabs, buffer chooser, etc.)
slide-14
SLIDE 14
slide-15
SLIDE 15

Editor Control Components (1/2)

slide-16
SLIDE 16

Editor Control Components (2/2)

slide-17
SLIDE 17

Generic Data Access (in Max/MSP)

  • Generic data container

mubu ... basic multi-buffer container imubu ... container with graphical user interface

  • Reference to a specific track

mubu.track ... optimized access to track data

* Generic record and play externals * JavaScript interface

slide-18
SLIDE 18

Analysis

  • FTM & Co prototype patch
  • loundness
  • power spectrum and spectral moments
  • MEL bands & MFCC coefs
  • YIN
  • loudness based segmentation

* PSOLA markers * Partials and harmonics * SuperVP

slide-19
SLIDE 19

Synthesis

  • ZsaZsa overlap-add engine
  • concatenative synthesis
  • granular synthesis

* PSOLA

* FFT-1 based additive synthesis * SuperVP

→ “spectral/hybrid concatenative synthesis”

slide-20
SLIDE 20

Data Models

  • KD-tree based k-nearest neighbors

* N-Grams? * Factor oracle? * ... what else?

slide-21
SLIDE 21

Current Release

  • Beta version of first set of externals
  • mubu ... container
  • imubu ... container with graphical interface
  • mubu.track ... optimized access to track data
  • mubu.knn ... k-nearest neighbors
  • mubu.concat~ ... concatenative synthesis
  • mubu.granular~ ... granular synthesis
slide-22
SLIDE 22

Future Developments

  • Finalize SDIF export
  • Integrate analysis and segmentation (real-time/offline)
  • Integrate spectral analysis/synthesis (real-time/offline)
  • PSOLA (ZsaZsa synthesis)
  • additive (FFT
  • 1 synthesis)
  • phase vocoder (SuperVP and simple)
  • Implement further data models
slide-23
SLIDE 23

Links

  • MuBu on the IMTR web pages

→ http://imtr.ircam.fr/imtr/MuBu

  • MuBu beta-test mailing list (subscription required)

→ http://listes.ircam.fr/wws/info/mubu-beta