Time-frequency multipliers for sound synthesis Ph. Depalle , R. - - PowerPoint PPT Presentation

Time-frequency multipliers for sound synthesis

Time-frequency multipliers for sound synthesis

• Ph. Depalle†, R. Kronland-Martinet‡ and B. Torr´

esani⋆

†: SPCL, McGill University, Montreal, Canada, ‡: LMA, Centre National de la Recherche Scientifique, Marseille, France ⋆: LATP, Universit´ e de Provence, Marseille, France

SPIE Symposium, San Diego, August 2007

Time-frequency multipliers for sound synthesis

Outline

Time-frequency sound synthesis: Generate sound signals from elementary, time-frequency localized atoms Implement sound transformations directly in the time-frequency domain

Time-frequency multipliers for sound synthesis

Outline

Time-frequency sound synthesis: Generate sound signals from elementary, time-frequency localized atoms Implement sound transformations directly in the time-frequency domain Goal: investigate the actual relevance of time-frequency multipliers and generalizations for sound synthesis and transformation.

Time-frequency multipliers for sound synthesis

1

Introduction

2

Time-frequency operator representation Time-frequency space Spreading function representation Gabor frames and multipliers Multiple Gabor multipliers

3

Gabor multiplier and multiple Gabor multiplier estimation The simple case Introducing time-frequency shifts MGM: masking pursuit

4

Applications to sound transformation

5

Conclusions

Time-frequency multipliers for sound synthesis Introduction

Time-frequency representations are known to be quite appropriate for representing audio signals: Audio signals often have good time-frequency localization properties The time-frequency plane allows one to easy model dependencies between coefficients We consider here the case of audio signal transformations, and their time-frequency implementation, in view of Sound synthesis from simple templates Various applications such as sound morphing, pitch shifting,... Here, we discuss the interest of time-frequency operator representation for quantifying timbre perceptual differences between sounds. An operator is represented by a (family of) time-frequency image(s), plus additional parameters.

Time-frequency multipliers for sound synthesis Introduction

Musical sound is described in terms of pitch and timbre: timbre is not very well defined, but several physics related cues are know to play significant role: The harmonicity: frequencies of partials may be multiple of a fundamental frequency (for example violin strings) or not (drums, piano strings,...). The harmonic content: for example, for wind instruments, boundary conditions determine the relative weight of even/odd harmonics (see example below) The time decay of the various harmonics, which is also related to the physics of the instrument The strength of the attack: i.e., the speed at which energy propagates across frequencies

Time-frequency multipliers for sound synthesis Introduction

Toy example

Sound signals composed of partials with partial dependent offset partial dependent initial amplitude partial dependent decay rate (exponential)

Time-frequency multipliers for sound synthesis Introduction

Toy example

Sound signals composed of partials with partial dependent offset partial dependent initial amplitude partial dependent decay rate (exponential) Magnitude of the Gabor Transform of toy example

Time-frequency multipliers for sound synthesis Introduction

Additive synthesis

x(t) =

• k

ak(t) cos(2πkf0t + ϕk) with (physics driven) ak(t) = cktγk exp{−t/τk} . simple model: τk = αβ−k

Time-frequency multipliers for sound synthesis Introduction

Real examples: Soprano saxophone and clarinet

Time (s) Frequency (Hz) Gabor Transform Magnitude of the Saxophone Tone 0.5 1 1.5 2 1000 2000 3000 4000 5000 6000 Time (s) Frequency (Hz) Gabor Transform Magnitude of the Clarinet Tone 0.5 1 1.5 2 1000 2000 3000 4000 5000 6000

Magnitude of the Gabor Transform of the saxophone tone (left) and of the clarinet tone (right).

Time-frequency multipliers for sound synthesis Introduction

Real examples: Soprano saxophone and clarinet

Time (s) Frequency (Hz) Gabor Transform Magnitude of the Saxophone Tone 0.5 1 1.5 2 1000 2000 3000 4000 5000 6000 Time (s) Frequency (Hz) Gabor Transform Magnitude of the Clarinet Tone 0.5 1 1.5 2 1000 2000 3000 4000 5000 6000

Magnitude of the Gabor Transform of the saxophone tone (left) and of the clarinet tone (right). Questions: Read the physical characteristics from time-frequency images (categorization) map (morph) saxophone into clarinet

Time-frequency multipliers for sound synthesis Introduction

Time-frequency morphing mask

Time (s) Frequency (Hz) Gabor Mask from a Saxophone to a Clarinet Tone (Magnitude) 0.5 1 1.5 2 1000 2000 3000 4000 5000 6000

Saxophone: Clarinet: Morphed Saxophone:

Time-frequency multipliers for sound synthesis Introduction

1

Introduction

2

Time-frequency operator representation Time-frequency space Spreading function representation Gabor frames and multipliers Multiple Gabor multipliers

3

Gabor multiplier and multiple Gabor multiplier estimation The simple case Introducing time-frequency shifts MGM: masking pursuit

4

Applications to sound transformation

5

Conclusions

Time-frequency multipliers for sound synthesis Time-frequency operator representation Time-frequency space

The time-frequency plane

Short time Fourier transform (STFT): associate to a (continuous time) signal x ∈ L2(R) the function Vgx ∈ L2(R2), defined by Vgx(b, ν) = ∞

−∞

x(t)e−2iπνtg(t − b) dt = x, g(b,ν), (1) where g is a fixed analysis window, and the atoms g(b,ν) are

• btained from g through time-frequency shifts π(b, ν) = MνTb

g(b,ν)(t) = exp{2iπνt}g(t − b) = [π(b, ν)]g(t) If g = 0, the STFT can be inverted in many ways: for any synthesis window h ∈ L2(R) such that g, h = 0, one has x(t) = 1 h, g ∞

−∞

∞

−∞

Vgx(b, ν)h(b,ν)(t) dbdν , (2) the equality holding in the strong L2(R) sense.

Time-frequency multipliers for sound synthesis Time-frequency operator representation Spreading function representation

Spreading function representation

Theorem (Time-frequency operator representation)

1 Let H ∈ H , the class of Hilbert-Schmidt operator on L2(R).

Then there exists a function η = ηH ∈ L2(R2), called the spreading function, such that H = ∞

−∞

∞

−∞

η(b, ν)π(b, ν) dbdν . (3) the integral being interpreted in the weak operator sense.

2 The relation η ∈ L2(R) ↔ H ∈ H extends to a Gelfand triple

isomorphism (S0(R), L2(R), S ′

0(R)) ↔ (B, H , B′).

Here, S0 denotes the space of functions whose STFT (with Gaussian window) is L1, S ′

0 its dual space, and B the space of

• perators that are bounded S ′

0 → S0.

Time-frequency multipliers for sound synthesis Time-frequency operator representation Spreading function representation

Spreading function representation (2)

The spreading function representation is closely related to the twisted convolution on the time-frequency plane, defined by (F♮G)(b, ν) =

• R2 F(b′, ν′)G(b − b′, ν − ν′)e−2iπb′(ν−ν′) db′dν′ .

(4)

Time-frequency multipliers for sound synthesis Time-frequency operator representation Spreading function representation

Spreading function representation (2)

The spreading function representation is closely related to the twisted convolution on the time-frequency plane, defined by (F♮G)(b, ν) =

• R2 F(b′, ν′)G(b − b′, ν − ν′)e−2iπb′(ν−ν′) db′dν′ .

(4) Theorem (Twisted convolution representation) Assume that g, h ∈ L2(R) are such that g, h = 1. Then H may be realized as a left twisted convolution in the time-frequency domain: for all x ∈ L2(R), VgHx = ηH♮Vgx . (5) (known as Weyl’s quantization in the theoretical physics literature)

Time-frequency multipliers for sound synthesis Time-frequency operator representation Spreading function representation

Remarks

An immediate consequence: the range of Vg is invariant under left twisted convolution.

Time-frequency multipliers for sound synthesis Time-frequency operator representation Spreading function representation

Remarks

An immediate consequence: the range of Vg is invariant under left twisted convolution. Digital signals are finite dimensional, and a corresponding finite-dimensional version may be derived. Unfortunately, such an expression is of poor practical interest. Numerical evaluation discrete twisted convolutions is extremely time consuming (O(N4) complexity). Subsampling the discrete twisted convolution results in very poor approximations. A better setting for discretizing such expressions is provided by (discrete) Gabor transforms.

Time-frequency multipliers for sound synthesis Time-frequency operator representation Gabor frames and multipliers

Gabor frames

Given lattice constants (b0, ν0), introduce the Gabor atoms gmn = g(mb0,nν0) , hmn = h(mb0,nν0) For suitable choices of g and the sampling constants, the Gabor atoms gmn form a frame, which implies that there exists synthesis windows h such that for all x ∈ L2(R), x =

• m,n

x, gmnhmn . Denote by Vg : x ∈ L2(R) − → Vgx , Vgx(m, n) = x, gmn the analysis operator

Time-frequency multipliers for sound synthesis Time-frequency operator representation Gabor frames and multipliers

Gabor multipliers

Consider a linear operator H ∈ H . When the spreading function ηH of H is sufficiently “concentrated” in the time-frequency space, the operator H can then be suitably approximated using the so-called Gabor multipliers, as follows Let again g, h ∈ L2(R), normalized so that g, h = 1 denote respectively the analysis and synthesis windows. Given any m ∈ ℓ∞(Z2), termed time-frequency transfer function (or mask) we define the associated Gabor multiplier Mm by Gabor transform, followed by multiplication by the mask, and inverse Gabor transform: Mm;g,hx(t) =

∞

• m,n=−∞

m(m, n)x, gmnhmn. (6)

Time-frequency multipliers for sound synthesis Time-frequency operator representation Multiple Gabor multipliers

Gabor multipliers (2)

Gabor multipliers are good at approximating operators whose spreading function is “concentrated enough”. Also Proposition (Best GM approximation [2, 3]) Assume that there exist real constants A, B such that 0 < A ≤

∞

• k,ℓ=−∞

|Vgh (t + k/ν0, ξ + ℓ/b0)|2 ≤ B < ∞ Then the best GM approximation (in Hilbert-Schmidt sense) of H ∈ H is defined by the time-frequency transfer function m whose discrete symplectic Fourier transform reads M(b, ν) =

• k,ℓ Vgh (b + k/ν0, ν + ℓ/b0) ηH (b + k/ν0, ν + ℓ/b0)
• k,ℓ |Vgh (b + k/ν0, ν + ℓ/b0)|2

(7)

Time-frequency multipliers for sound synthesis Time-frequency operator representation Multiple Gabor multipliers

Multiple Gabor Multipliers

When the spreading function ηH of H is not well enough localized, it is still possible to seek approximations as sums of Gabor multipliers. The Multiple Gabor Multipliers (MGM), introduced in [2, 3] are linear combinations of Gabor multipliers, with fixed analysis window, and variable synthesis windows (with different time-frequency localizations) h(j) and masks mj, and are defined as follows: M =

• Mmj;g,h(j)

(8) Optimal MGM approximations for Hilbert-Schmidt operators can be obtained as in the GM case.

Time-frequency multipliers for sound synthesis Time-frequency operator representation Multiple Gabor multipliers

Multiple Gabor Multipliers (2)

Theorem Let g, h ∈ S0 as well as b0, ν0 be given. Furthermore, let h(m,n) = π( m

ν0 , n b0 )h. Set

Amn(b, ν) =

• k,ℓ e2iπm[ν−ℓ/ν0] V (b − k/ν0, ν − ℓ/b0)

× V (b − (k − m)/ν0, ν − (ℓ − n)/b0), Then the symplectic Fourier transform of the vector of masks m(m,n) of the optimal MGM approximation of H ∈ H has to satisfy a.e. M(b, ν)♮A(b, ν) = B(b, ν) , (9) where B is given by Bj0(b, ν) =

• k,ℓ

η(b + k/ν0, ν + ℓ/b0)V j0(b + k/ν0, ν + ℓ/b0).

Time-frequency multipliers for sound synthesis Time-frequency operator representation Multiple Gabor multipliers

1

Introduction

2

Time-frequency operator representation Time-frequency space Spreading function representation Gabor frames and multipliers Multiple Gabor multipliers

3

Gabor multiplier and multiple Gabor multiplier estimation The simple case Introducing time-frequency shifts MGM: masking pursuit

4

Applications to sound transformation

5

Conclusions

Time-frequency multipliers for sound synthesis Gabor multiplier and multiple Gabor multiplier estimation The simple case

Gabor multiplier estimation

Suppose we have the input signal x0, and an observation x1 of the form x1(t) = Mg,g;mx0(t) + ǫ1(t) , from which we try to estimate the mask m. It is natural to seek a solution by minimizing Φ[m] = x1 − Mg,g;mx02 + λm2 , (10) where the Lagrange parameter λ ∈ R+ is introduced in order to control the norm of m. The latter quantity reads Φ[m] = V ∗

g (Vgx1 − mVgx0) 2 + λm2 ,

and is not obvious to minimize, because V ∗

g is not one to one.

Time-frequency multipliers for sound synthesis Gabor multiplier and multiple Gabor multiplier estimation The simple case

Gabor multiplier estimation (2)

The geometrical picture

Time-frequency multipliers for sound synthesis Gabor multiplier and multiple Gabor multiplier estimation The simple case

Gabor multiplier estimation (2)

The geometrical picture As an alternative, one uses the proxy defined by the minimizer of Ψ[m] = Vgx1 − mVgx02 + λm2 , (11) which reads m = Vgx0 Vgx1 |Vgx0|2 + λ . (12)

Time-frequency multipliers for sound synthesis Gabor multiplier and multiple Gabor multiplier estimation The simple case

Gabor multiplier estimation (3)

When several realizations of the output signal (with varying amplitude) are available, with a unique input: xk = akMg,g;mx0 + ǫk , Similar considerations yield m(m, n) = K

k=1 akVgxk(m, n)

• Vgx0(m, n)

K

k=1 |ak|2 |Vgx0|2

+ Kλ , (13) and ak =

• m,n m(m, n)Vgx0(m, n)Vgxk(m, n)
• m,n |m(m, n)|2 |Vgx0(m, n)|2

(14)

Time-frequency multipliers for sound synthesis Gabor multiplier and multiple Gabor multiplier estimation Introducing time-frequency shifts

Time-frequency shifted multipliers

Assume now the following model x1(t) = Mℓν0Tkb0Mg,g;mx0(t) + ǫ1(t) , which naturally leads to the optimization of Φ[m, k, ℓ] = x1 − Mℓν0Tkb0Mg,g;mx02 + λm2 .

Time-frequency multipliers for sound synthesis Gabor multiplier and multiple Gabor multiplier estimation Introducing time-frequency shifts

Time-frequency shifted multipliers

Assume now the following model x1(t) = Mℓν0Tkb0Mg,g;mx0(t) + ǫ1(t) , which naturally leads to the optimization of Φ[m, k, ℓ] = x1 − Mℓν0Tkb0Mg,g;mx02 + λm2 . Assume first that k and ℓ are known. Then one is back to the same problem as before, replacing x1 with T−kb0M−ℓν0x1. Assume now that the mask m is known. One has to solve min

k,ℓ Φ[m, πkℓ] ⇔ max k,ℓ ℜ(x1, πkℓMg,g;mx0) .

Then by noticing that x1, πkℓy = Vyx1(k, ℓ) the search of minima with respect to k, ℓ amounts to a search for the maxima of the cross-ambiguity function Vyx1.

Time-frequency multipliers for sound synthesis Gabor multiplier and multiple Gabor multiplier estimation MGM: masking pursuit

Multiple Gabor Multiplier: Masking pursuit

Gabor multipliers are often not sufficient for correctly approximating operators of interest in sound analysis, and one has to turn to Multiple Gabor multipliers. Difficulties: The best MGM approximation of a given operator with known spreading function requires the inversion of a system of twisted convolutions, which may turn out to be a difficult problem in real world situations. In addition, in the problem under consideration here, the MGM has to be estimated from data, which makes the task more complex. Alternative: Masking Pursuit, an algorithm which estimates iteratively Gabor multipliers composed with time-frequency shifts.

Time-frequency multipliers for sound synthesis Gabor multiplier and multiple Gabor multiplier estimation MGM: masking pursuit

Multiple Gabor Multiplier: Masking pursuit (2)

Let x0 and x1 denote respectively the input and output signals as before. Initialization: Set r(0) = x1. Iteration: for n = 0, ..Nmax − 1,

Estimate a mask mn+1 and time-frequency shifts (kn+1, ℓn+1) from x0 and residual r (n), using the above approach. Update the residual r (n+1) = r (n) − πkn+1ℓn+1Mmn+1;g,gx0

This yields an estimate H ≈

Nmax−1

• n=0

πknℓnMmn;g,g for a MGM approximation of the operator

Time-frequency multipliers for sound synthesis Gabor multiplier and multiple Gabor multiplier estimation MGM: masking pursuit

1

Introduction

2

Time-frequency operator representation Time-frequency space Spreading function representation Gabor frames and multipliers Multiple Gabor multipliers

3

Gabor multiplier and multiple Gabor multiplier estimation The simple case Introducing time-frequency shifts MGM: masking pursuit

4

Applications to sound transformation

5

Conclusions

Time-frequency multipliers for sound synthesis Applications to sound transformation

Attack transformation

Consider the simple example of two amplitude modulated sinusoids, with different amplitude functions. The goal is to assess the ability of Gabor multipliers to describe the attack, which plays a key role in sound perception. The attack of the source sound starts at time N0 = 0.01 ∗ SR and consists in a cosine-shape modulation for a duration N = 0.05 ∗ SR. The attack of the target sound is a simple Heaviside function which discontinuity is located at time N.

Time-frequency multipliers for sound synthesis Applications to sound transformation

Attack transformation (2)

Time (s) Frequency (Hz) Gabor Transform Magnitude of the Source 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 1000 2000 3000 4000 5000 6000 7000 8000 9000 Time (s) Frequency (Hz) Gabor Transform Magnitude of the Target 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 1000 2000 3000 4000 5000 6000 7000 8000 9000

Gabor Transform magnitude of source (left) and target sounds (right).

Time (s) Frequency (Hz) Gabor Mask between Source and Target 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 1000 2000 3000 4000 5000 6000 7000 8000 9000

Magnitude of the Gabor Mask.

Time-frequency multipliers for sound synthesis Applications to sound transformation

Attack transformation (3)

Time (s) Frequency (Hz) Gabor Transform Magnitude of the Source 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 1000 2000 3000 4000 5000 6000 7000 8000 9000 Time (s) Frequency (Hz) Gabor Transform Magnitude of the Target 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 1000 2000 3000 4000 5000 6000 7000 8000 9000

Gabor Transform magnitude of source (left) and target sounds (right).

Time (s) Frequency (Hz) Gabor Mask between Source and Target 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 1000 2000 3000 4000 5000 6000 7000 8000 9000

Magnitude of the Gabor Mask.

Time-frequency multipliers for sound synthesis Applications to sound transformation

Masking pursuit

The third example deals with the general case of the multiple Gabor multipliers and illustrates the masking pursuit algorithm. Source signal: single amplitude modulated sinusoid, at frequency 440Hz, with linear attack. Target signal: is a superposition of three equal-amplitude sinusoids, at respective frequencies 1000, 2000, and 3000Hz, and linear attack. In the next slide: the top row shows the magnitudes of the first three estimated masks, and the second row shows the corresponding magnitudes of Gabor transform of residual target.

Time-frequency multipliers for sound synthesis Applications to sound transformation

Masking pursuit (2)

Time (s) Frequency (Hz) Gabor Transform Magnitude of Signal u1 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 500 1000 1500 2000 2500 3000 3500 4000 4500 Time (s) Frequency (Hz) Gabor Transform Magnitude of Signal u2 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 500 1000 1500 2000 2500 3000 3500 4000 4500 Time (s) Frequency (Hz) Gabor Transform Magnitude of Signal u3 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 500 1000 1500 2000 2500 3000 3500 4000 4500 Time (s) Frequency (Hz) Gabor Transform Magnitude of Signal r1 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 500 1000 1500 2000 2500 3000 3500 4000 4500 Time (s) Frequency (Hz) Gabor Transform Magnitude of Signal r2 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 500 1000 1500 2000 2500 3000 3500 4000 4500 Time (s) Frequency (Hz) Gabor Transform Magnitude of Signal r3 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 500 1000 1500 2000 2500 3000 3500 4000 4500

Iteration 1 Iteration 2 Iteration 3

Time-frequency multipliers for sound synthesis Applications to sound transformation

1

Introduction

2

Time-frequency operator representation Time-frequency space Spreading function representation Gabor frames and multipliers Multiple Gabor multipliers

3

Gabor multiplier and multiple Gabor multiplier estimation The simple case Introducing time-frequency shifts MGM: masking pursuit

4

Applications to sound transformation

5

Conclusions

Time-frequency multipliers for sound synthesis Conclusions

Conclusions

Gabor multipliers, time-frequency shifted Gabor multipliers and multiple Gabor multipliers are sensible tools for sound transformation and design. Gabor multiplier and time-frequency shifted Gabor multiplier estimation may be performed efficiently. The global estimation of a multiple Gabor multiplier is more problematic; the masking pursuit algorithm provides a fairly simple alternative. All numerical calculations were done using the Linear Time-Frequency Analysis Toolbox developed by Peter Søndergaard [9].

Time-frequency multipliers for sound synthesis Conclusions

Conclusions

Gabor multipliers, time-frequency shifted Gabor multipliers and multiple Gabor multipliers are sensible tools for sound transformation and design. Gabor multiplier and time-frequency shifted Gabor multiplier estimation may be performed efficiently. The global estimation of a multiple Gabor multiplier is more problematic; the masking pursuit algorithm provides a fairly simple alternative. All numerical calculations were done using the Linear Time-Frequency Analysis Toolbox developed by Peter Søndergaard [9].

Thanks for your attention !

Time-frequency multipliers for sound synthesis Conclusions

• R. Carmona, W. Hwang, and B. Torr´

esani. Practical Time-Frequency Analysis: continuous wavelet and Gabor transforms, with an implementation in S, volume 9 of Wavelet Analysis and its Applications. Academic Press, San Diego, 1998.

• M. D¨
• rfler and B. Torr´

esani. Spreading function representation of operators and Gabor multiplier approximation. In Sampling Theory and Applications (SAMPTA’07), Thessaloniki, June 2007.

• M. D¨
• rfler and B. Torr´

esani. On the time-frequency representation of operators and generalized Gabor multiplier approximations. preprint (2007), submitted

Time-frequency multipliers for sound synthesis Conclusions

H.G. Feichtinger and T. Strohmer Eds., Gabor analysis and algorithms, Series in Applied and Numerical Harmonic Analysis, Birkh¨ auser Boston Inc. (1998).

• H. G. Feichtinger and K. Nowak.

A first survey of Gabor multipliers. In H. G. Feichtinger and T. Strohmer, editors, Advances in Gabor Analysis, Boston, 2002. Birkhauser.

• K. Gr¨
• chenig.

Foundations of Time-Frequency Analysis. Birkha¨ user, Boston, 2001.

• F. Hlawatsch and G. Matz.

Linear time-frequency filters. In B. Boashash, editor, Time-Frequency Signal Analysis and Processing: A Comprehensive Reference, page 466:475, Oxford (UK), 2003. Elsevier.

Time-frequency multipliers for sound synthesis Conclusions

• M. Kahrs and K. Brandenbourg, Applications of Digital Signal

Processing to Audio and Acoustics, Kluwer Academic Press, Dortrecht, The Netherland, (1998).

• P. Søndergaard, Ltfat, the Linear Time-Frequency Analysis

Toolbox, available online at http://sourceforge.net/projects/ltfat

• N. H. Fletcher and T.D. Rossing. The physics of musical

instruments, Springer-Verlag, New-York, (1991).