Audio declipping Matthieu Kowalski Univ Paris-Sud L2S (GPI) - PowerPoint PPT Presentation

Introduction Direct model Inverse problem Numerical results Audio declipping Matthieu Kowalski Univ Paris-Sud L2S (GPI) Matthieu Kowalski Audio declipping 1 / 22

Introduction Direct model Inverse problem Numerical results Introduction 1 Direct model 2 Inverse problem 3 Numerical results 4 Matthieu Kowalski Audio declipping 2 / 22

Introduction Direct model Inverse problem Numerical results Audio Declipping Original signal: 1 0.5 0 -0.5 -1 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Time (s) Matthieu Kowalski Audio declipping 3 / 22

Introduction Direct model Inverse problem Numerical results Audio Declipping Clipped signal: 1 0.5 0 -0.5 -1 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Time (s) Matthieu Kowalski Audio declipping 4 / 22

Introduction Direct model Inverse problem Numerical results Audio Declipping 1 0.8 0.6 0.4 0.2 0 −0.2 −0.4 −0.6 −0.8 −1 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Goal: Can we get a good estimation of the original signal (blue) from the clipped one (red) ? Matthieu Kowalski Audio declipping 5 / 22

Introduction Direct model Inverse problem Numerical results Reliable vs Unreliable coeff. y r = M r y Original s (unknown) Observation y 10000000000000000   01000000000000000   00100000000000000     00001000000000000     M r = 00000100000000000     00000010000000000     00000000010000000 M r     00000000001000000     00000000000000100   Degradation 00000000000000010 00010000000000000   00000001000000000   00000000100000000 M m   M m =   00000000000100000     00000000000010000     00000000000001000   00000000000000001 Reliable data y m = M m y Unreliable data Missing data to be estimated Matthieu Kowalski Audio declipping 6 / 22

Introduction Direct model Inverse problem Numerical results Reliable vs Unreliable coeff. Reliable samples: y r = M r y 1 0.5 0 -0.5 -1 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Time (s) Matthieu Kowalski Audio declipping 7 / 22

Introduction Direct model Inverse problem Numerical results Reliable vs Unreliable coeff. Unreliable (clipped) samples: y m = M m y 1 0.5 0 -0.5 -1 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Time (s) Matthieu Kowalski Audio declipping 8 / 22

Introduction Direct model Inverse problem Numerical results Reliable vs Unreliable coeff. Reliable + Unreliable (clipped) samples: 1 0.5 0 -0.5 -1 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Time (s) Matthieu Kowalski Audio declipping 9 / 22

Introduction Direct model Inverse problem Numerical results Audio inpainting: forward problem [A. Adler, V. Emiya et Al ] We have then: y r = M r y = M r s where s ∈ R N is the unknown “clean” signal; y r ∈ R M are the “reliable” sample of the observed signal M r ∈ R M × N is the matrix of the reliable support of x we can also define the ”missing” samples as y m = M m y = M m s Matthieu Kowalski Audio declipping 10 / 22

Introduction Direct model Inverse problem Numerical results Inverse problem: data term Using the reliable coefficients, we must have y r = M r s where M r select the reliable samples. We can use a simple ℓ 2 loss 1 2 � y r − M r s � 2 s = argmin ˆ 2 s We must take the clipped samples into account Matthieu Kowalski Audio declipping 11 / 22

Introduction Direct model Inverse problem Numerical results Inverse problem: clipping constraints For audio declipping, we can add the following constraint 1 2 � y r − M r s � 2 ˆ s = argmin 2 s M m + Φ α > θ clip s.t. M m − Φ α < − θ clip where M m + (resp. M m − ) select the positive (resp. negative) clipped samples. θ clip is the clip threshold (here θ clip = 0 . 2) Problem: infinite solutions! We must add some constraints on s Matthieu Kowalski Audio declipping 12 / 22

Introduction Direct model Inverse problem Numerical results Audio declipping: use a dictionnary Let Φ a dictionnary such that: s = Φ α where α are sparse synthesis coefficients Audio signal: use the short time Fourier transform � s ( t ) = Φ α = α n , f ϕ n , f ( t ) n , f 4 2.2x 10 2 1.8 1.6 Frequency (Hz) 1.4 1.2 1 0.8 0.6 0.4 0.2 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 Time (s) Matthieu Kowalski Audio declipping 13 / 22

Introduction Direct model Inverse problem Numerical results Inverse problem: a constrained sparse problem Using the dictionnary Φ + sparsity 1 2 � y r − M r Φ α � 2 α = argmin ˆ 2 + λ � α � 1 s M m + Φ α > θ clip s.t. M m − Φ α < − θ clip where M m + (resp. M m − ) select the positive (resp. negative) clipped samples. θ clip is the clip threshold (here θ clip = 0 . 2) ˆ s = ˆ α Problems: the proximity operator has no closed form Cannot use simple algorithms such as (F)ISTA Matthieu Kowalski Audio declipping 14 / 22

Introduction Direct model Inverse problem Numerical results Rewrite the constraints Idea: use a ℓ 2 loss on the clipped samples if the constraint is not respected If y m ( t ) > θ clip then L ( θ clip − y m ( t )) = 0 y m ( t ) < θ clip If ˆ else L ( θ clip − y m ( t )) = ( θ clip − y m ( t )) 2 25 20 15 10 5 0 -5 -4 -3 -2 -1 0 1 2 3 4 5 Matthieu Kowalski Audio declipping 15 / 22

Introduction Direct model Inverse problem Numerical results Rewrite the constraints The squared hinge loss: L ( θ clip − y m ) = [ θ clip − y m ] 2 + ( θ clip − y m ( t )) 2 ( − θ clip + y m ( t )) 2 � � = + + + t : y m ( t ) > 0 t : y m ( t ) < 0 = [ θ clip − M m Φ α ] 2 + 25 20 15 10 5 0 -5 -4 -3 -2 -1 0 1 2 3 4 5 Matthieu Kowalski Audio declipping 16 / 22

Introduction Direct model Inverse problem Numerical results Audio declipping: (convex unconstrained) inverse problem We consider the following unconstrained convex problem: 1 2 + 1 2[ θ clip − M m Φ α ] 2 2 � y r − M r Φ α � 2 α = argmin + + λ � α � 1 α which is under the form f 1 ( α ) + f 2 ( α ) with f 1 Lipschitz-differentiable and f 2 semi-convex. We can apply (relaxed)-ISTA directly ! Matthieu Kowalski Audio declipping 17 / 22

Introduction Direct model Inverse problem Numerical results FISTA for declipping Matthieu Kowalski Audio declipping 18 / 22

Introduction Direct model Inverse problem Numerical results Thresolding operators Matthieu Kowalski Audio declipping 19 / 22

Introduction Direct model Inverse problem Numerical results Numerical results Speech @ 16kHz Music @ 16kHz 18 11 L 10 Average SNR m Improvement Average SNR m Improvement 16 EW WGL 9 14 PEW 8 HT 12 OMP 7 10 6 5 8 4 6 3 4 2 2 1 0 0 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 Clipping Level Clipping Level Average SNR miss for 10 speech (left) and music (right) signals over different clipping levels and operators. Neighborhoods extend 3 and 7 coefficients in time for speech and music signals, respectively. Matthieu Kowalski Audio declipping 20 / 22

Introduction Direct model Inverse problem Numerical results Numerical results: zoom on reconstructions 0.8 0.6 0.4 Amplitude 0.2 0 Original −0.2 PEW EW −0.4 L WGL −0.6 HT OMP −0.8 4230 4235 4240 4245 4250 4255 Time in ms Declipped music signal using different operators for clip level θ clip = 0 . 2 using the Lasso, WGL, EW, PEW, HT, and OMP operators. Neighborhood size for WGL and PEW was 7. Matthieu Kowalski Audio declipping 21 / 22

Introduction Direct model Inverse problem Numerical results Original Vs clipped Vs declipped Signal 0.8 0.6 0.4 0.2 0 − 0.2 − 0.4 − 0.6 − 0.8 0 1 2 3 4 5 1 0.5 0 − 0.5 − 1 0 1 2 3 4 5 Matthieu Kowalski Audio declipping 22 / 22