Cosparse Low-dimensional Signal Modelling Mehrdad Yaghoobi, In collaboration with: Sangnam Nam, Remi Gribonval, and Mike E. Davies I V N E U R S E I H T Y T H O G F R E U D B I N Workshop on Sparsity, Compressed Sensing and Applications, Centre for Digital Music, Queen Mary University of London, UK November 5th, 2012 Cosparse Low-dimensional Signal Modelling 1 / 22
Low-dimensional Signal Models Union of Subspaces Model Point Cloud Model Smooth Manifold Model Cosparse Low-dimensional Signal Modelling 2 / 22
Some Applications of Low-dimensional Models Denoising: Embedding: Inpainting: Contaminating with Embedding to a lower- Masking the signal with dimensional space R M , noise, M , i.e. z = My . z = y + n . using Φ, i.e. z = Φ y . Denoising: Recovering: Inpainting: y ∗ = argmin θ ∈U � z − θ � 2 y ∗ = argmin θ ∈U � z − Φ θ � 2 y ∗ = argmin θ ∈U � z − θ � 2 M Cosparse Low-dimensional Signal Modelling 3 / 22
Sparse Synthesis Model Each subspace can be interpreted as the span of a small number of atoms. This model can be used to represent the signals. It has been used for many applications, particularly for regularising inverse problems. Synthesis Sparsity Model The signal y follows the model, if there exists an (overcomplete) dictionary D ∈ R n × p , p ≥ n , such that y can be represented by, y = Dx , where � x � 0 = k and k is called the sparsity of y , in D . Cosparse Low-dimensional Signal Modelling 4 / 22
Cosparse Analysis Model Each subspace can be interpreted by a set of normal vectors. This low-dimensional model is based on constraining the possible signals. It has often been used for denoising. Analysis Model The signal y follows the model, if there exists a (linear) analysis operator Ω ∈ R a × n , a ≥ n that sparsifies y , z = Ω y . � z � 0 = a − q , where q > 0 is called the co-sparsity of y , with respect to Ω. Cosparse Low-dimensional Signal Modelling 5 / 22
Dictionary Learning A set of exemplars Y =[ y 1 . . . y i . . . y L ] is given. The goal is to find a suitable dictionary for the synthesis sparse representation of training samples. The dictionary is often learned by minimising an objective which simultaneously sparsify the solution and reduce the fidelity of sparse representation. Learning Formulation The dictionary can be learned by minimising an objective based on X and D , X , D � X � 1 + λ 2 � Y − DX � 2 min F s . t . D ∈ D The constraint D is necessary to resolve the scale ambiguity. Cosparse Low-dimensional Signal Modelling 6 / 22
Analysis Operator Learning (AOL) Similarly, the set Y = [ y 1 . . . y i . . . y L ] is given. The goal is to find an analysis operator Ω such that � Ω � Y � 0 is small, where � Y is close to Y . When the noiseless exemplars are available, the AOL is easier, as we do not need to find � Y . The objective is non-smooth ⇒ not suitable for optimisation with variational techniques. Formulation The learned operator can be found by minimising the sparsity promoting operator, Y � 1 + θ � Ω � 2 � � Y − Y � 2 min F s . t . Ω ∈ C Ω , b Y where C is a constraint to exclude the trivial solutions, e.g. Ω= 0 . Cosparse Low-dimensional Signal Modelling 7 / 22
Insufficient Constraints Row norm constraints Row norm + full rank Tight frame constraints constraints ∀ i , � ω i � 2 = c It resolves the issue in a A randomly perturbed Ω complete setting. In the Rank one Ω 1 is found by from Ω 1 , i.e. row overcomplete cases, it repeating the best normalised Ω 1 + N , has a includes zero-padded (almost) orthogonal direction ω ∗ to columns full rank and it is still not orthobases. suitable. of Y . Cosparse Low-dimensional Signal Modelling 8 / 22
Proposed Constraint Uniform Normalised Tight Frame (UNTF): Definition: C = { Ω ∈ R n × m : Ω T Ω = I & ∀ i � ω i � 2 = � m n } Pros and Cons: Zero-padded orthobases are not UNTF. There exist some practical methods to project onto the TF and the UN manifolds. However, there is no analytical way to find the projection onto the UNTF! There is no easy way to find the global optimum, using C as the constraint. Cosparse Low-dimensional Signal Modelling 9 / 22
Cosparse Analysis Operator Learning Algorithm Iterative Analysis Operator Learning Algorithm Y � 1 + θ � Ω � 2 � � Y − Y � 2 min s . t . Ω ∈ C . F Ω , b Y Solving by alternating minimisation technique. Optimisation based on Ω : Minimisation of a convex objective subject to the intersection of two manifolds ⇒ a variant of projected subgradient algorithm is a good candidate. Optimisation based on � Y : a convex program. → Douglas-Rachford Splitting (DRS) technique was used to efficiently solve the program. Here, algorithm usually converges after a few number of alternating minimisation. Cosparse Low-dimensional Signal Modelling 10 / 22
Projected Subgradient Algorithm for AOL Projected Subgradient Type Algorithm for AOL 1: initialisation: k = 1, K max , Ω [0] = 0 , Ω [1] = Ω in , γ, ǫ ≪ 1 2: while ǫ ≤ � Ω [ k ] − Ω [ k − 1] � F and k ≤ K max do Ω G = ∂ f (Ω [ k ] ) 3: � � �� Ω [ k +1] = P UN Ω [ k ] − γ Ω G P TF 4: k = k + 1 5: 6: end while 7: output: Ω out = Ω [ k − 1] . Cosparse Low-dimensional Signal Modelling 11 / 22
AOL for the Piecewise Constant Images Finding an Ω for the image patches of size 8 × 8. A 512 × 512 Shepp-Logan phantom image was used as the training image in a noiseless setting. N = 16384 image patches was randomly chosen from the training image. A pseudo-random UNTF operator Ω 0 ∈ R 128 × 64 was used as the initial operator and K max was 100,000. Cosparse Low-dimensional Signal Modelling 12 / 22
AOL for the Piecewise Constant Images Original Operator Learned Operator Cosparse Low-dimensional Signal Modelling 13 / 22
Issues with the Projected Subgradient Algorithm: Some Proposed Relaxations No analytical way to project onto UNTF → no convergence proof. Projection onto TF needs a full SVD calculation → expensive implementation and non-scalable algorithm. ℓ 1 term is not differentiable → slow convergence of the projected subgradient algorithm. Relaxed AOL Formulation Relaxing the objective: using a convex, but differentiable sparsity 1 constraint g (Ω Y ), where g is an entrywise function defined as, g ( x ) = | x | − s ln(1 + | x | / s ) , s ∈ R + , s ≪ 1 Relaxing the constraint: using quartic constraints 2 � � 2 ≤ ǫ UN , � Ω T Ω − I � 2 ω T i ω i − m F ≤ ǫ TF and ∀ i ∈ [1 , n ] n Cosparse Low-dimensional Signal Modelling 14 / 22
Relaxed Analysis Operator Learning Relaxed Analysis Operator Learing Formulation An unconstrained objective is generate by using two Lagrange multipliers γ and λ : � � 2 � f ( Ω ) = g ( ΩY ) + γ F + λ i ω i − m 4 � Ω T Ω − I � 2 ω T . 4 n i f ( Ω ) is differentiable and it would also be convex, if we restrict its domain to C c = { Ω : Ω T Ω − I � 0 , ∀ i , ( ω T i ω i − m n ) ≥ 0 } . Gradient Descent Algorithm for AOL A variable step-size gradient descent, with line search, can be used to minimise f ( Ω ), where the gradient of f can easily be found by: � � � � � � �� T Z i , j i ω i − m Y T + γ ΩΩ T − I ω T ∇ f = Ω + λ ω i s + | Z i , j | n i i , j Z := ΩY Cosparse Low-dimensional Signal Modelling 15 / 22
Relaxation of the Constraints (a) (b) (c) 3.5 1 1 0.999 0.9 3 0.998 0.8 2.5 0.997 0.7 0.996 0.6 2 0.995 0.5 1.5 0.994 0.4 0.993 0.3 1 0.2 0.992 0.5 0.1 0.991 0 0 0.99 5 10 15 20 2 4 6 8 10 12 14 16 5 10 15 20 Learning an Ω ∈ R 24 × 16 from Y ∈ R 16 × 576 q = 10 cosparse exemplars. Sorted ℓ 2 norms of the rows of learned operator with the TF constraint (left). Singular values of the learned operator with the UN constraint (middle). Normalised inner-products between the rows of the synthetic ideal operator and the corresponding rows in the learned operator (right). Cosparse Low-dimensional Signal Modelling 16 / 22
An Operator for the Face Images: Setting Learning an Ω for the image face patches from the Yale face database. L = 16384, 8 × 8 image patches were randomly selected from different faces. The noise-aware and noiseless AOL methods were used for operator learning. Cosparse Low-dimensional Signal Modelling 17 / 22
An Operator for Face Images: Cosparsity Comparison The analysis coefficients z = Ωy and cosparsities were calculated, using Ω 0 , Ω AOL and Ω NAAOL . (a) 80 20 N − || Ω 0 y || 0 = 0 Cosparsity with operator learned with AOL Cosparsity with operator learned with NAAOL 70 0 60 −20 0 20 40 60 80 100 120 (b) 50 20 N − || Ω y || 0 = 1 Cosparsity 40 0 30 −20 0 20 40 60 80 100 120 (c) 20 10 N − || Ω y ~ || 0 = 27 10 0 0 −10 0 20 40 60 80 100 120 0 50 100 150 200 250 Sample number Cosparse Low-dimensional Signal Modelling 18 / 22
Learned Operator Original Operator Learned Operator Cosparse Low-dimensional Signal Modelling 19 / 22
Recommend
More recommend
