Structured Sparsity in Gabor Analysis Dominik Fuchs University of - - PowerPoint PPT Presentation

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Structured Sparsity

in Gabor Analysis Dominik Fuchs

University of Vienna Faculty of Mathematics

WS 2012/13

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Contents

1

Motivation

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Contents

1

Motivation

2

Sparse Regularization Problem Penalty Measure

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Contents

1

Motivation

2

Sparse Regularization Problem Penalty Measure

3

Thresholding and Iterative Algorithms Thresholding (F)ISTA

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Contents

1

Motivation

2

Sparse Regularization Problem Penalty Measure

3

Thresholding and Iterative Algorithms Thresholding (F)ISTA

4

Persistence and Neighborhood Neighborhood Empirical Wiener

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Contents

1

Motivation

2

Sparse Regularization Problem Penalty Measure

3

Thresholding and Iterative Algorithms Thresholding (F)ISTA

4

Persistence and Neighborhood Neighborhood Empirical Wiener

5

Matlab Examples

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Structure

1

Motivation

2

Sparse Regularization Problem Penalty Measure

3

Thresholding and Iterative Algorithms Thresholding (F)ISTA

4

Persistence and Neighborhood Neighborhood Empirical Wiener

5

Matlab Examples

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Narural Signals

‘Natural’ signals often yield unwanted noise disturbing the original sound.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Narural Signals

‘Natural’ signals often yield unwanted noise disturbing the original sound. Consider:

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Narural Signals

‘Natural’ signals often yield unwanted noise disturbing the original sound. Consider:

• ‘air rustle’ or even backgroundnoise in microphoned signals

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Narural Signals

‘Natural’ signals often yield unwanted noise disturbing the original sound. Consider:

• ‘air rustle’ or even backgroundnoise in microphoned signals
• directly recorded music with electric instruments

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Narural Signals

‘Natural’ signals often yield unwanted noise disturbing the original sound. Consider:

• ‘air rustle’ or even backgroundnoise in microphoned signals
• directly recorded music with electric instruments
• clipping

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Microphoned Signals

Example 1: There could be some background noise.

Microphoned Signal

Time (s) Frequency (Hz) 2 4 6 8 0.5 1 1.5 2 x 10

4

−70 −60 −50 −40 −30 −20 −10 10 Time (s) Frequency (Hz) 2 4 6 8 0.5 1 1.5 2 x 10

4

−70 −60 −50 −40 −30 −20 −10 10

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Electric Instruments

Example 2a: Recordings with bad input can cause some noise.

Electric Guitar

Time (s) Frequency (Hz) 1 2 3 4 5 0.5 1 1.5 2 x 10

4

−70 −60 −50 −40 −30 −20 −10 10 Time (s) Frequency (Hz) 1 2 3 4 5 0.5 1 1.5 2 x 10

4

−80 −70 −60 −50 −40 −30 −20 −10

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Electric Instruments

Example 2b: Or even more noise.

Flanger

Time (s) Frequency (Hz) 1 2 3 4 5 6 7 8 0.5 1 1.5 2 x 10

4

−70 −60 −50 −40 −30 −20 −10 10 Time (s) Frequency (Hz) 1 2 3 4 5 6 7 8 0.5 1 1.5 2 x 10

4

−80 −70 −60 −50 −40 −30 −20 −10

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Clipping

Example 3: Too loud input signals can lead to clipping.

Clipped Signal

Time (s) Frequency (Hz) 0.5 1 1.5 2 2.5 3 3.5 0.5 1 1.5 2 x 10

4

−50 −40 −30 −20 −10 10 20 30 Time (s) Frequency (Hz) 0.5 1 1.5 2 2.5 3 3.5 0.5 1 1.5 2 x 10

4

−60 −50 −40 −30 −20 −10 10 20

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Structure

1

Motivation

2

Sparse Regularization Problem Penalty Measure

3

Thresholding and Iterative Algorithms Thresholding (F)ISTA

4

Persistence and Neighborhood Neighborhood Empirical Wiener

5

Matlab Examples

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Problem

We will start by formulating the problem.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Problem

We will start by formulating the problem. For our natural signal y, we get y = f + e where f is the clean/wanted signal and e the additional noise.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Discrepancy

For checking the variance of our signal under the synthesis we define the discrepancy

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Discrepancy

For checking the variance of our signal under the synthesis we define the discrepancy Definition The Discrepancy is defined by ∆(c) := 1 2y − Φc2

2

with synthesis operator Φ : Hc → Hs, Φ = (ϕ1, ..., ϕγ, ...), signal y and coefficients c ∈ Hc. We need to find coefficients, s.t. ∆(c) of the data y and the image

• f c is minimized.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Lagrangian

Problem:

• solution is not unique
• not continuously dependent on the data

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Lagrangian

Problem:

• solution is not unique
• not continuously dependent on the data

We need to take additional constraints on the coefficients into account. Definition The regularized functional called Lagrangian is defined by L(c) := Ly,λ(c) := ∆(c) + λΨ(c) where Ψ : Hc → R+

0 is the so called penalty measure and λ > 0

the Lagrange-multiplier resp. sparsity level.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Lagrangian

Our aim is to seek ˆ c ∈ Hc such that ˆ c = argmin

c

L(c)

This is the first general formulation of our problem for sparse regularization!

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Non-convex Problem

For sparsity we want to minimize the number of non-zero coefficients c0 := #{cγ : cγ = 0}, i.e. Ψ = c0

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Non-convex Problem

For sparsity we want to minimize the number of non-zero coefficients c0 := #{cγ : cγ = 0}, i.e. Ψ = c0 !! Problem not solvable in finite time !!

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Non-convex Problem

For sparsity we want to minimize the number of non-zero coefficients c0 := #{cγ : cγ = 0}, i.e. Ψ = c0 !! Problem not solvable in finite time !! It was shown that ℓ1-minimization uniquely recovers the ℓ0-solution. By using the ℓ1-norm instead of the ℓ0-norm we get the convex minimization problem →

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

LASSO

ˆ c = argmin

c

1 2y − Φc2

2 + λc1

• This Problem is known as the so called

LASSO - least absolute shrinkage and selection operator. An equivalent formulation was given in the field of statistics with the name Basis Pursuit Denoising.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Mixed Norms

Viewing problem in Gabor-analysis ⇒ atoms ordered along two dimensions

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Mixed Norms

Viewing problem in Gabor-analysis ⇒ atoms ordered along two dimensions = ⇒ makes sense to split indices into groups and members

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Mixed Norms

Viewing problem in Gabor-analysis ⇒ atoms ordered along two dimensions = ⇒ makes sense to split indices into groups and members Realizable by replacing our penalty by a weighted mixed norm.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Mixed Norms

Definition (Weighted Mixed Norm) Let Γ be a doubly labelled index set and K, J1, J2, ..., Jk, ... be countable index sets such that Γk := {(k, j) : j ∈ Jk} ∀k ∈ K we have Γ = ˙

• k∈KΓk.

Let w = (wγ)γ∈Γ be a positive sequence of weights. The weighted mixed norm ℓw,p,q on Hc for 1 ≤ p, q < ∞ is defined by cw,p,q :=   

• k∈K

 

j∈Jk

wk,j|ck,j|p  

q/p

 

1/q

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Lasso with Mixed Norms

By reformulating the Lagrangian by setting Ψ(c) = 1

qcq w,p,q we

get Lw,p,q(c) := 1 2y − Φc2

2 + 1

q cq

w,p,q.

and the resulting sparse recovery problem argmin

c

Lw,p,q(c).

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Problem Penalty Measure

Lasso with Mixed Norms

The mixed norms will consider in the further context and their problems names are:

• ℓw,2,1 ..... Group Lasso (GL)
• ℓw,1,2 ..... Elitist Lasso (EL)

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Thresholding (F)ISTA

Structure

1

Motivation

2

Sparse Regularization Problem Penalty Measure

3

Thresholding and Iterative Algorithms Thresholding (F)ISTA

4

Persistence and Neighborhood Neighborhood Empirical Wiener

5

Matlab Examples

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Thresholding (F)ISTA

Thresholding

We introduce a new thresholding operator Sξ that fullfills ˆ c = Sw,p,q(a) = argmin

c

1 2c − a2

2 + 1

q cq

w,p,q

• for any a ∈ Hc. In fact a = Φ∗y.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Thresholding (F)ISTA

Thresholding

Definition (Gerneralized Thresholding Operator) For z, w ∈ Hc, wγ > δ > 0 and a non-negative function ξ = ξγ,w : Hc → [0, ∞], the generalized thresholding operator is defined component-wise by Sξ(zγ) := zγ(1 − ξγ,w(z))+ where b ∈ R, b+ := max(b, 0). ξ is called the threshold function.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Thresholding (F)ISTA

Threshold functions for Mixed Norms

• p = q = 1 : ξw(zγ) = wγ

|zγ|

(Lasso)

• p = q = 2 : ξw(zγ) =

wγ 1+wγ

(Tykhonov Regularization)

• p = 2, q = 1; wk,j = wk∀k, j : ξw(zk,j) =

√wk,j zk2

(Group-Lasso)

• p = 1, q = 2 : ξw(zk,j) =

wk,j 1+Wwk | |zk|wk |zk,j|

(Elitist-Lasso)

where Wwk := Jk

jk =1 w2 k,jk , and |zk|wk = Jk jk =1 wk,jk |zk,jk | and for any k, jk

is a sequence of indices such that rk,jk :=

|zk,jk | wk,jk

is decreasing in jk, and Jk is the quantity verifying rk,Jk +1 ≤

Jk +1

• jk =1

w2

k,jk (rk,jk − rk,Jk +1)

and rk,Jk >

Jk

• jk =1

w2

k,jk (rk,jk − rk,Jk ) Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Thresholding (F)ISTA

ISTA

For Φ : Hc → Hs bounded we get a sequence χ(c) = Sw(c + Φ∗(y − Φc)) that converges to a minimizer of the Lagrangian L.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Thresholding (F)ISTA

ISTA

For Φ : Hc → Hs bounded we get a sequence χ(c) = Sw(c + Φ∗(y − Φc)) that converges to a minimizer of the Lagrangian L. This leads us directly to the soft-thresholding algorithm (ISTA), also called thresholded Landweber iteration: cn+1 = (χn+1(c0))n+1 = Sw(cn + Φ∗(y − Φcn))

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Thresholding (F)ISTA

ISTA to FISTA

Since the ISTA - Algorithm with the iteration step cn = S(bn) with bn = cn−1 + Φ∗(y − Φcn−1) converges very slowly we want to improve out algorithm. We handle this by modify the choice of bn by a linear combination of cn and cn−1.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Thresholding (F)ISTA

FISTA - Algorithm

Algorithm:

For S = Sw,p,q, let c0 = b1 ∈ Hs and t1 = 1. Do cn = S(bn + Φ∗(y − Φbn)) tn+1 = 1

2(1 +

• 1 + 4t2

n)

bn+1 = cn +

• tn−1

tn+1

• (cn − cn−1)

Until convergence

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Neighborhood Empirical Wiener

Structure

1

Motivation

2

Sparse Regularization Problem Penalty Measure

3

Thresholding and Iterative Algorithms Thresholding (F)ISTA

4

Persistence and Neighborhood Neighborhood Empirical Wiener

5

Matlab Examples

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Neighborhood Empirical Wiener

Persistence Expansion

Often there may occur some sharp cut-off’s. The idea of ” smoothing”the coefficients leads to the definition of neighborhood weights.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Neighborhood Empirical Wiener

Persistence Expansion

Often there may occur some sharp cut-off’s. The idea of ” smoothing”the coefficients leads to the definition of neighborhood weights. By convolving a threshold function ξ with a neighborhood-smoothing functional η.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Neighborhood Empirical Wiener

Neighborhood

Definition (Time-Frequency Neightborhood) For the countable index set Λ the time-frequency neighborhood weights are defined as the non-negative sequences vγ = vγ(˜ γ) ≥ 0 ∀γ, ˜ γ ∈ Λ which fulfill the following properties: vγ2 = 1,

˜ γ

v˜

γ(γ) ≤ C < ∞ ∀γ, vγ(γ) > 0 ∀γ

Nγ := supp(vγ) = {˜ γ ∈ Γ : vγ(˜ γ) > 0} is called the time-frequency neighborhood of γ.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Neighborhood Empirical Wiener

Neighborhood-smoothing functional

Definition For given neughborhood weights vγ, let the neighborhood-smoothing functional η : Hc → R+

0 is defined

component-wise by η(cγ) :=  

˜ γ∈Γ

vγ(˜ γ)|c˜

γ|2

 

1/2

For c ∈ Hc, we set η(c) := (η(cγ))γ∈Γ).

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Neighborhood Empirical Wiener

Operators

We have now a couple of operators: ξL = ξ1,1 − Lasso (L) ξGL = ξ2,1 − Group − Lasso (GL) ξEL = ξ1,2 − Elitist − Lasso (EL) ξWGL = ξ∗

1,1 = ξL ∗ ηN − windowed Group Lasso (WGL)

ξPGL = ξ∗

2,1 = ξGL ∗ ηN − persistent Group Lasso (PGL)

ξPEL = ξ∗

1,2 = ξEL ∗ ηN − persistent Elitist Lasso (PEL)

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Neighborhood Empirical Wiener

Threshold Selection

Until now the shrinkage level λ has been linear. Of course the idea

• f changing this value during the iteration would come up. For this

the theory of fusing structured sparsity with empirical Wiener filtering has been introduced.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Neighborhood Empirical Wiener

Threshold Selection

Until now the shrinkage level λ has been linear. Of course the idea

• f changing this value during the iteration would come up. For this

the theory of fusing structured sparsity with empirical Wiener filtering has been introduced. We skip the derivation and just consider the relation to our

• perators so far.

To read more about this theory, take a look into the article Kai Siedenburg, ” Persistent Empirical Wiener estimation with adaptive threshold selection for audio denoising” , ¨ OFAI, 2012

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples Neighborhood Empirical Wiener

Persistent Empirical Wiener

By taking our Soft-thresholding operator Sξ and taking the second power of the threshold function ξ we get the Empirical Wiener

• perator

SEW := Sα

ξ (zγ) := zγ(1 − ξγ,w(z)α)+

where α = 2.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Structure

1

Motivation

2

Sparse Regularization Problem Penalty Measure

3

Thresholding and Iterative Algorithms Thresholding (F)ISTA

4

Persistence and Neighborhood Neighborhood Empirical Wiener

5

Matlab Examples

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Tools

In the following examples we see some applications performed with the StrucAudioToolbox.

(Download: http://homepage.univie.ac.at/monika.doerfler/StrucAudio.html)

The images we will see, are visualizations of the ‘active’ coefficients, not to be confounded with the spectogram.

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Denoising: Flanger

The noisy ’Flanger’ from above denoised with an PEW.

Denoised Original Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Denoising: Microphoned signal

The microphoned signal with backgroundnoise from a party denoised with an GL and grouplabel in the frequency domain and 5 iteration steps.

Denoised Original

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Denoising: Microphoned signal

By setting the grouplabel in time, we might get reconstucted signals like this.

Denoised Original

Dominik Fuchs Structured Sparsity

Motivation Sparse Regularization Thresholding and Iterative Algorithms Persistence and Neighborhood Matlab Examples

Multi-layer Decomposition

For

Musical Clock with an addition generated noise we can perform a

seperation into tonal and transients parts and add them together afterwards.

Transient Multilayer Original Tonal

Dominik Fuchs Structured Sparsity