Cosparse Low-dimensional Signal Modelling Mehrdad Yaghoobi, In - - PowerPoint PPT Presentation

Cosparse Low-dimensional Signal Modelling

Mehrdad Yaghoobi,

In collaboration with: Sangnam Nam, Remi Gribonval, and Mike E. Davies

T H E U N I V E R S I T Y O F E D I N B U R G H

Workshop on Sparsity, Compressed Sensing and Applications, Centre for Digital Music, Queen Mary University of London, UK

November 5th, 2012

1 / 22 Cosparse Low-dimensional Signal Modelling

Low-dimensional Signal Models

Union of Subspaces Model Point Cloud Model Smooth Manifold Model

2 / 22 Cosparse Low-dimensional Signal Modelling

Some Applications of Low-dimensional Models

Denoising: Contaminating with noise, z = y + n. Denoising: y∗ = argminθ∈U z − θ2 Embedding: Embedding to a lower- dimensional space RM, using Φ, i.e. z = Φy. Recovering: y∗ = argminθ∈U z−Φθ2 Inpainting: Masking the signal with M, i.e. z = My. Inpainting: y∗ = argminθ∈U z−θ2

M

3 / 22 Cosparse Low-dimensional Signal Modelling

Sparse Synthesis Model

Each subspace can be interpreted as the span

• f a small number of atoms.

This model can be used to represent the signals. It has been used for many applications, particularly for regularising inverse problems.

Synthesis Sparsity Model The signal y follows the model, if there exists an (overcomplete) dictionary D ∈ Rn×p, p ≥ n, such that y can be represented by, y = Dx, where x0 = k and k is called the sparsity of y, in D.

4 / 22 Cosparse Low-dimensional Signal Modelling

Cosparse Analysis Model

Each subspace can be interpreted by a set of normal vectors. This low-dimensional model is based on constraining the possible signals. It has often been used for denoising.

Analysis Model The signal y follows the model, if there exists a (linear) analysis

• perator Ω ∈ Ra×n, a ≥ n that sparsifies y,

z = Ωy. z0 = a − q, where q > 0 is called the co-sparsity of y, with respect to Ω.

5 / 22 Cosparse Low-dimensional Signal Modelling

Dictionary Learning

A set of exemplars Y=[y1 . . . yi . . . yL] is given. The goal is to find a suitable dictionary for the synthesis sparse representation of training samples. The dictionary is often learned by minimising an

• bjective which simultaneously sparsify the

solution and reduce the fidelity of sparse representation.

Learning Formulation The dictionary can be learned by minimising an objective based on X and D, min

X,D X1 + λ

2 Y − DX2

F s. t. D ∈ D

The constraint D is necessary to resolve the scale ambiguity.

6 / 22 Cosparse Low-dimensional Signal Modelling

Analysis Operator Learning (AOL)

Similarly, the set Y = [y1 . . . yi . . . yL] is given. The goal is to find an analysis operator Ω such that Ω Y0 is small, where Y is close to Y. When the noiseless exemplars are available, the AOL is easier, as we do not need to find Y. The objective is non-smooth ⇒ not suitable for

• ptimisation with variational techniques.

Formulation The learned operator can be found by minimising the sparsity promoting operator, min

Ω,b Y

Ω Y1 + θ 2 Y − Y2

F s. t. Ω ∈ C

where C is a constraint to exclude the trivial solutions, e.g. Ω=0.

7 / 22 Cosparse Low-dimensional Signal Modelling

Insufficient Constraints

Row norm constraints ∀i, ωi2 = c Rank one Ω1 is found by repeating the best (almost) orthogonal direction ω∗ to columns

• f Y.

Row norm + full rank constraints A randomly perturbed Ω from Ω1, i.e. row normalised Ω1 + N, has a full rank and it is still not suitable. Tight frame constraints It resolves the issue in a complete setting. In the

• vercomplete cases, it

includes zero-padded

• rthobases.

8 / 22 Cosparse Low-dimensional Signal Modelling

Proposed Constraint

Uniform Normalised Tight Frame (UNTF): Definition: C = {Ω ∈ Rn×m : ΩTΩ = I & ∀i ωi2 = m

n }

Pros and Cons:

Zero-padded orthobases are not UNTF. There exist some practical methods to project onto the TF and the UN manifolds. However, there is no analytical way to find the projection onto the UNTF! There is no easy way to find the global optimum, using C as the constraint.

9 / 22 Cosparse Low-dimensional Signal Modelling

Cosparse Analysis Operator Learning Algorithm

Iterative Analysis Operator Learning Algorithm min

Ω,b Y

Ω Y1 + θ 2 Y − Y2

F

• s. t.

Ω ∈ C. Solving by alternating minimisation technique.

Optimisation based on Ω: Minimisation of a convex objective subject to the intersection of two manifolds ⇒ a variant of projected subgradient algorithm is a good candidate. Optimisation based on Y: a convex program. → Douglas-Rachford Splitting (DRS) technique was used to efficiently solve the program. Here, algorithm usually converges after a few number of alternating minimisation.

10 / 22 Cosparse Low-dimensional Signal Modelling

Projected Subgradient Algorithm for AOL

Projected Subgradient Type Algorithm for AOL

1: initialisation: k = 1, Kmax, Ω[0] = 0, Ω[1] = Ωin, γ, ǫ ≪ 1 2: while ǫ ≤ Ω[k] − Ω[k−1]F and k ≤ Kmax do 3:

ΩG = ∂f (Ω[k])

4:

Ω[k+1] = PUN

• PTF
• Ω[k] − γΩG
• 5:

k = k + 1

6: end while 7: output: Ωout = Ω[k−1].

11 / 22 Cosparse Low-dimensional Signal Modelling

AOL for the Piecewise Constant Images

Finding an Ω for the image patches of size 8 × 8. A 512 × 512 Shepp-Logan phantom image was used as the training image in a noiseless setting. N = 16384 image patches was randomly chosen from the training image. A pseudo-random UNTF

• perator Ω0 ∈ R128×64 was

used as the initial operator and Kmax was 100,000.

12 / 22 Cosparse Low-dimensional Signal Modelling

AOL for the Piecewise Constant Images

Original Operator Learned Operator

13 / 22 Cosparse Low-dimensional Signal Modelling

Issues with the Projected Subgradient Algorithm: Some Proposed Relaxations

No analytical way to project onto UNTF → no convergence proof. Projection onto TF needs a full SVD calculation → expensive implementation and non-scalable algorithm. ℓ1 term is not differentiable → slow convergence of the projected subgradient algorithm.

Relaxed AOL Formulation

1

Relaxing the objective: using a convex, but differentiable sparsity constraint g(ΩY), where g is an entrywise function defined as, g(x) = |x| − s ln(1 + |x|/s), s ∈ R+, s ≪ 1

2

Relaxing the constraint: using quartic constraints ΩTΩ − I2

F ≤ ǫTF and

• ωT

i ωi − m n

2 ≤ ǫUN, ∀i ∈ [1, n]

14 / 22 Cosparse Low-dimensional Signal Modelling

Relaxed Analysis Operator Learning

Relaxed Analysis Operator Learing Formulation An unconstrained objective is generate by using two Lagrange multipliers γ and λ: f (Ω) = g(ΩY) + γ 4ΩTΩ − I2

F + λ

4

• i
• ωT

i ωi − m

n 2 .

f (Ω) is differentiable and it would also be convex, if we restrict its domain to Cc = {Ω : ΩTΩ − I 0, ∀i, (ωT

i ωi − m n ) ≥ 0}.

Gradient Descent Algorithm for AOL A variable step-size gradient descent, with line search, can be used to minimise f (Ω), where the gradient of f can easily be found by: ∇f =

• Zi,j

s + |Zi,j|

• i,j

YT + γ

• ΩΩT − I
• Ω + λ
• ωi
• ωT

i ωi − m

n T

i

Z := ΩY

15 / 22 Cosparse Low-dimensional Signal Modelling

Relaxation of the Constraints

2 4 6 8 10 12 14 16 0.5 1 1.5 2 2.5 3 3.5 (b) 5 10 15 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 (a) 5 10 15 20 0.99 0.991 0.992 0.993 0.994 0.995 0.996 0.997 0.998 0.999 1 (c)

Learning an Ω ∈ R24×16 from Y ∈ R16×576 q = 10 cosparse exemplars. Sorted ℓ2 norms of the rows of learned operator with the TF constraint (left). Singular values of the learned operator with the UN constraint (middle). Normalised inner-products between the rows of the synthetic ideal

• perator and the corresponding rows in the learned operator (right).

16 / 22 Cosparse Low-dimensional Signal Modelling

An Operator for the Face Images: Setting

Learning an Ω for the image face patches from the Yale face database. L = 16384, 8 × 8 image patches were randomly selected from different faces. The noise-aware and noiseless AOL methods were used for operator learning.

17 / 22 Cosparse Low-dimensional Signal Modelling

An Operator for Face Images: Cosparsity Comparison

The analysis coefficients z = Ωy and cosparsities were calculated, using Ω0, ΩAOL and ΩNAAOL.

20 40 60 80 100 120 −20 20 N − ||Ω0 y ||0 = 0 (a) 20 40 60 80 100 120 −20 20 N − ||Ω y ||0 = 1 (b) 20 40 60 80 100 120 −10 10 N − ||Ω y~||0 = 27 (c) 50 100 150 200 250 10 20 30 40 50 60 70 80 Sample number Cosparsity Cosparsity with operator learned with AOL Cosparsity with operator learned with NAAOL

18 / 22 Cosparse Low-dimensional Signal Modelling

Learned Operator

Original Operator Learned Operator

19 / 22 Cosparse Low-dimensional Signal Modelling

Face Images Denoising: TV v.s. Learned Operator

TV operator for comparison. Two different regularisation parameters, λ = 0.3 & 0.1 .

50 100 150 200 250 300 350 400 450 500 50 100 150 Cosparsity with the learned operator 50 100 150 200 250 300 350 400 450 500 50 100 150 Cosparsity with the Total Variation operator 50 100 150 200 250 300 350 400 450 500 −20 20 40 Differences between cosparsities Image patch number

(a) (b) (c) (d) (e) (f) 20 / 22 Cosparse Low-dimensional Signal Modelling

Conclusion and Future Work

Conclusion:

The constrained analysis operator learning is a useful technique to find a suitable analysis operator. The proposed constraint can be relaxed to reduce the complexity of the optimisation algorithm, while including some approximately UNTF operators. The simulation results emphasis on the fact that we should use the correct analysis operator, i.e. TV or oscillatory operators. The convergence of the relaxed AOL is guaranteed, as its objective has a bounded curvature and its sublevel set is compact.

Future Work:

◮ Investigating the local identifiability of operators in this framework. ◮ More investigations on the structures of the learned operators.

21 / 22 Cosparse Low-dimensional Signal Modelling

Thanks for your attention.