TIME-DEPENDENT PARAMETRIC AND HARMONIC TEMPLATES IN NON-NEGATIVE - - PowerPoint PPT Presentation

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion

TIME-DEPENDENT PARAMETRIC AND HARMONIC TEMPLATES IN NON-NEGATIVE MATRIX FACTORIZATION

13th International Conference on Digital Audio Effects Romain Hennequin, Roland Badeau and Bertrand David

Telecom ParisTech

September 8, 2010

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 1/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion

Introduction

Musical spectrograms decomposition (on a basis of notes) Decomposition based on Non-negative Matrix Factorization (NMF) Spectrogram models are introduced into decomposition methods:

parametric harmonic atoms makes it possible to model slight pitch variations

Potential applications:

Multipitch estimation/transcription Source separation

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 2/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion

Sommaire

1 Non negative-Matrix Factorization 2 Spectrogram model 3 Decomposition of musical spectrograms

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 3/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Principle Issues Proposed solution

Contents

1 Non negative-Matrix Factorization Principle Issues Proposed solution 2 Spectrogram model 3 Decomposition of musical spectrograms

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 4/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Principle Issues Proposed solution

Principle of NMF

Low-rank approximation: V ≈ ˆ V = WH ˆ Vft =

R

• r=1

WfrHrt

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 5/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Principle Issues Proposed solution

Issues with NMF

Pitch variations Low-rank approximation does not permit to model variations over time, such as slight pitch variations (vibrato. . . ).

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 6/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Principle Issues Proposed solution

Issues with NMF

time (frames) frequency (kHz) Original spectrogram 50 100 150 1 2 3 4 5 time (frames) frequency (kHz) NMF spectrogram R = 1 50 100 150 1 2 3 4 5

Note with vibrato: Decomposition with a single atom.

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 7/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Principle Issues Proposed solution

Issues with NMF

time (frames) frequency (kHz) Original spectrogram 50 100 150 1 2 3 4 5 time (frames) frequency (kHz) NMF spectrogram R = 3 50 100 150 1 2 3 4 5

Note with vibrato: Decomposition with 3 atoms.

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 8/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Principle Issues Proposed solution

Proposed solution

What does an atom look like in a musical spectrogram? In a musical spectrogram most of the (non-percussive) elements are instruments notes which are generally harmonic tones. Parameters of interest are generally the fundamental frequency of these tones, and the shape of the amplitudes of the harmonics. Proposed method: parametric model of spectrogram with harmonic atoms.

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 9/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Parametric spectrogram Parametric atoms Algorithm

Contents

1 Non negative-Matrix Factorization 2 Spectrogram model Parametric spectrogram Parametric atoms Algorithm 3 Decomposition of musical spectrograms

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 10/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Parametric spectrogram Parametric atoms Algorithm

Parametric spectrogram

Time-varying atoms in NMF: ˆ Vft =

R

• r=1

WfrHrt → ˆ Vft =

R

• r=1

Wθrt

fr Hrt

θrt is a time-varying parameter associated to each atom. In this paper, θrt is the fundamental frequency f rt

• f each atom.

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 11/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Parametric spectrogram Parametric atoms Algorithm

Parametric atoms

Parametric harmonic atom construction Wf rt

fr = nh(f rt

0 )

• k=1

akg(f − kf rt

0 )

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 12/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Parametric spectrogram Parametric atoms Algorithm

Parametric spectrogram

Hypotheses of the model The harmonic part of notes is supposed to be stationary within an analysis frame. Interferences between harmonics are supposed to be negligible. Classical hypothesis of NMF about positive summation of parts.

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 13/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Parametric spectrogram Parametric atoms Algorithm

Algorithm

Parametric spectrogram ˆ Vft =

R

• r=1

nh

• k=1

akg(f − kf rt

0 )

• W

f rt fr

hrt Learnt parameters A divergence between V and ˆ V is to be minimized w.r.t.: f rt

0 : the fundamental frequency of each atom at each frame

ak: the amplitudes of harmonics (Atoms share the same set of amplitudes) hrt: the activation of each atom at each frame Cost function: C(f rt

0 , ak, hrt) = D(Vft|ˆ

Vft)

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 14/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Parametric spectrogram Parametric atoms Algorithm

Algorithm

Minimization Global optimization w.r.t. f rt is impossible (numerous local minima in C). ⇒ one atom is introduced for each MIDI note. Optimization thus becomes local (fine estimate of f rt

0 ).

Minimization achieved with multiplicative update rules. Remark The proposed method is no longer a rank-reduction method but still reduces the data dimension.

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 15/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Decomposition Improvement Estimated frequency Real signals

Contents

1 Non negative-Matrix Factorization 2 Spectrogram model 3 Decomposition of musical spectrograms Decomposition Improvement Estimated frequency Real signals

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 16/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Decomposition Improvement Estimated frequency Real signals

Decomposition of a synthetic spectrogram

Time (frame) Frequency (kHz) Original power spectrogram

50 100 150 200 250 300 1 2 3 4 5 −5 5 10 15 20 25 30 35 40

Spectrogram of the first bars of JS Bach’s first prelude played by a synthesizer.

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 17/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Decomposition Improvement Estimated frequency Real signals

Obtained decomposition

Frames Semitones 50 100 150 200 250 300 10 20 30 40 50 60 70 −35 −30 −25 −20 −15

Activations for each MIDI note.

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 18/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Decomposition Improvement Estimated frequency Real signals

Obtained decomposition

Decomposition Notes appear at the right place with decreasing amplitudes Numerous atoms activated at onset time Notes activated at octave, twelfth and double octave of the right note (note with many common partials).

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 19/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Decomposition Improvement Estimated frequency Real signals

Improvement

Onset A few standard NMF atoms can be used to model onsets: ˆ Vft =

R

• r=1

Wθrt

fr Hrt + K

• k=1

AfkBkt Octaves, twelfths. . . Add constraints to the cost function: Sparsity constraints on activations Decorrelation constraints (between activations of octaves. . . ) Smoothness constraints on amplitudes

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 20/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Decomposition Improvement Estimated frequency Real signals

Obtained decomposition

Frames Semitones 50 100 150 200 250 300 10 20 30 40 50 60 70 −30 −25 −20 −15 −10

Activations for each MIDI note.

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 21/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Decomposition Improvement Estimated frequency Real signals

Time/frequency representation

Frames Semitones

20 40 60 80 100 120 140 160 14 16 18 20 22 24 26 28 30 32 34 20 25 30 35 40

Activations centered on estimated frequency for each MIDI note: vibrato appears.

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 22/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Decomposition Improvement Estimated frequency Real signals

Issues with real signals

Frames Semitones 50 100 150 200 250 300 10 20 30 40 50 60 70 −30 −25 −20 −15 −10

Activations for each MIDI note. (Piano sound)

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 23/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion Decomposition Improvement Estimated frequency Real signals

Issues with real signals

Issues The model of amplitudes of harmonics is quite rough Issues with onsets and octaves are more important Noisy components (breath. . . ) Some instruments are not perfectly harmonic (piano. . . )

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 24/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion

Conclusion

Summary New way of decomposing musical spectrograms with slight pitch variations in constituting elements. Parametric thus flexible model. Perspectives Improve decomposition to make it more adapted to real data: Better modeling of harmonic amplitudes Supervised learning of amplitudes Better onset and noise modeling

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 25/26

Introduction Non negative-Matrix Factorization Spectrogram model Decomposition of musical spectrograms Conclusion

Conclusion

Any questions?

Romain Hennequin, Roland Badeau and Bertrand David Time-dependent parametric templates in NMF - slide 26/26