Audio declipping Matthieu Kowalski Univ Paris-Sud L2S (GPI) - - PowerPoint PPT Presentation

Introduction Direct model Inverse problem Numerical results

Audio declipping

Matthieu Kowalski

Univ Paris-Sud L2S (GPI)

Matthieu Kowalski Audio declipping 1 / 22

Introduction Direct model Inverse problem Numerical results

1

Introduction

2

Direct model

3

Inverse problem

4

Numerical results

Matthieu Kowalski Audio declipping 2 / 22

Introduction Direct model Inverse problem Numerical results

Audio Declipping

Original signal:

0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Time (s)

• 1
• 0.5

0.5 1 Matthieu Kowalski Audio declipping 3 / 22

Introduction Direct model Inverse problem Numerical results

Audio Declipping

Clipped signal:

0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Time (s)

• 1
• 0.5

0.5 1 Matthieu Kowalski Audio declipping 4 / 22

Introduction Direct model Inverse problem Numerical results

Audio Declipping

Goal:

0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1

Can we get a good estimation of the original signal (blue) from the clipped one (red) ?

Matthieu Kowalski Audio declipping 5 / 22

Introduction Direct model Inverse problem Numerical results

Reliable vs Unreliable coeff.

Unreliable data Observation y Missing data to be estimated Original s (unknown) Degradation Mm Mr yr = Mry ym = Mmy

Mm =

         00010000000000000 00000001000000000 00000000100000000 00000000000100000 00000000000010000 00000000000001000 00000000000000001         

Mr =

               10000000000000000 01000000000000000 00100000000000000 00001000000000000 00000100000000000 00000010000000000 00000000010000000 00000000001000000 00000000000000100 00000000000000010               

Reliable data

Matthieu Kowalski Audio declipping 6 / 22

Introduction Direct model Inverse problem Numerical results

Reliable vs Unreliable coeff.

Reliable samples: yr = Mry

0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Time (s)

• 1
• 0.5

0.5 1 Matthieu Kowalski Audio declipping 7 / 22

Introduction Direct model Inverse problem Numerical results

Reliable vs Unreliable coeff.

Unreliable (clipped) samples: ym = Mmy

0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Time (s)

• 1
• 0.5

0.5 1 Matthieu Kowalski Audio declipping 8 / 22

Introduction Direct model Inverse problem Numerical results

Reliable vs Unreliable coeff.

Reliable + Unreliable (clipped) samples:

0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Time (s)

• 1
• 0.5

0.5 1 Matthieu Kowalski Audio declipping 9 / 22

Introduction Direct model Inverse problem Numerical results

Audio inpainting: forward problem [A. Adler, V. Emiya et Al]

We have then: yr = Mry = Mrs where s ∈ RN is the unknown “clean” signal; yr ∈ RM are the “reliable” sample of the observed signal Mr ∈ RM×N is the matrix of the reliable support of x we can also define the ”missing” samples as ym = Mmy = Mms

Matthieu Kowalski Audio declipping 10 / 22

Introduction Direct model Inverse problem Numerical results

Inverse problem: data term

Using the reliable coefficients, we must have yr = Mrs where Mr select the reliable samples. We can use a simple ℓ2 loss ˆ s = argmin

s

1 2yr − Mrs2

2

We must take the clipped samples into account

Matthieu Kowalski Audio declipping 11 / 22

Introduction Direct model Inverse problem Numerical results

Inverse problem: clipping constraints

For audio declipping, we can add the following constraint ˆ s = argmin

s

1 2yr − Mrs2

2

s.t. Mm+Φα > θclip Mm−Φα < −θclip where Mm+ (resp. Mm−) select the positive (resp. negative) clipped samples. θclip is the clip threshold (here θclip = 0.2) Problem: infinite solutions! We must add some constraints on s

Matthieu Kowalski Audio declipping 12 / 22

Introduction Direct model Inverse problem Numerical results

Audio declipping: use a dictionnary

Let Φ a dictionnary such that: s = Φα where α are sparse synthesis coefficients Audio signal: use the short time Fourier transform s(t) = Φα =

• n,f

αn,f ϕn,f (t)

Time (s) Frequency (Hz) 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2x 10

4

Matthieu Kowalski Audio declipping 13 / 22

Introduction Direct model Inverse problem Numerical results

Inverse problem: a constrained sparse problem

Using the dictionnary Φ + sparsity ˆ α = argmin

s

1 2yr − MrΦα2

2 + λα1

s.t. Mm+Φα > θclip Mm−Φα < −θclip where Mm+ (resp. Mm−) select the positive (resp. negative) clipped samples. θclip is the clip threshold (here θclip = 0.2) ˆ s = ˆ α Problems: the proximity operator has no closed form Cannot use simple algorithms such as (F)ISTA

Matthieu Kowalski Audio declipping 14 / 22

Introduction Direct model Inverse problem Numerical results

Rewrite the constraints

Idea: use a ℓ2 loss on the clipped samples if the constraint is not respected If y m(t) > θclip then L(θclip − y m(t)) = 0 If ˆ y m(t) < θclip else L(θclip − y m(t)) = (θclip − y m(t))2

• 5
• 4
• 3
• 2
• 1

1 2 3 4 5 5 10 15 20 25

Matthieu Kowalski Audio declipping 15 / 22

Introduction Direct model Inverse problem Numerical results

Rewrite the constraints

The squared hinge loss: L(θclip − ym) = [θclip − ym]2

+

=

• t:y m(t)>0

(θclip − y m(t))2

+ +

• t:y m(t)<0

(−θclip + y m(t))2

+

= [θclip − MmΦα]2

+

• 5
• 4
• 3
• 2
• 1

1 2 3 4 5 5 10 15 20 25

Matthieu Kowalski Audio declipping 16 / 22

Introduction Direct model Inverse problem Numerical results

Audio declipping: (convex unconstrained) inverse problem

We consider the following unconstrained convex problem: α = argmin

α

1 2yr − MrΦα2

2 + 1

2[θclip − MmΦα]2

+ + λα1

which is under the form f1(α) + f2(α) with f1 Lipschitz-differentiable and f2 semi-convex. We can apply (relaxed)-ISTA directly !

Matthieu Kowalski Audio declipping 17 / 22

Introduction Direct model Inverse problem Numerical results

FISTA for declipping

Matthieu Kowalski Audio declipping 18 / 22

Introduction Direct model Inverse problem Numerical results

Thresolding operators

Matthieu Kowalski Audio declipping 19 / 22

Introduction Direct model Inverse problem Numerical results

Numerical results

0.2 0.4 0.6 0.8 1 2 4 6 8 10 12 14 16 18

Clipping Level Average SNRm Improvement Speech @ 16kHz

L EW WGL PEW HT OMP

0.2 0.4 0.6 0.8 1 1 2 3 4 5 6 7 8 9 10 11

Clipping Level Average SNRm Improvement Music @ 16kHz

Average SNRmiss for 10 speech (left) and music (right) signals over different clipping levels and operators. Neighborhoods extend 3 and 7 coefficients in time for speech and music signals, respectively.

Matthieu Kowalski Audio declipping 20 / 22

Introduction Direct model Inverse problem Numerical results

Numerical results: zoom on reconstructions

4230 4235 4240 4245 4250 4255 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8

Time in ms Amplitude

Original PEW EW L WGL HT OMP

Declipped music signal using different operators for clip level θclip = 0.2 using the Lasso, WGL, EW, PEW, HT, and OMP operators. Neighborhood size for WGL and PEW was 7.

Matthieu Kowalski Audio declipping 21 / 22

Introduction Direct model Inverse problem Numerical results

Original Vs clipped Vs declipped Signal

1 2 3 4 5 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 2 3 4 5 −1 −0.5 0.5 1 Matthieu Kowalski Audio declipping 22 / 22