Dictionary Learning for Graph Signals 236862 Introduction to Sparse - - PowerPoint PPT Presentation

Dictionary Learning for Graph Signals

Yael Yankelevsky 22.12.2019

236862 – Introduction to Sparse and Redundant Representations

joint work with

• Prof. Michael Elad

Dictionary Learning for Graph Signals By: Yael Yankelevsky

The Sparseland Model

2

Model assumption: All data vectors are linear combinations of FEW (𝑈 ≪ 𝑂) atoms from 𝐄

= Dictionary Learning:

Dictionary Learning for Graph Signals By: Yael Yankelevsky

≈

… …

X Y D

Initialize Dictionary Sparse Coding

Using OMP

Dictionary Update

Atom-by-atom + coeffs.

For the j-th atom:

K-SVD Algorithm Overview [Aharon et al. ’06]

3

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Energy Networks Biological Networks Transportation Networks Social Networks Meshes & Point Clouds

Data is often structured…

4

Dictionary Learning for Graph Signals By: Yael Yankelevsky

5

What happens for non-conventionally structured signals? Can dictionary learning work well for such signals as well? The general idea: Model the underlying structure as a graph and incorporate it in the dictionary learning algorithm

Dictionary Learning for Graph Signals By: Yael Yankelevsky

We are given a graph:

• The 𝑗𝑢ℎ node is characterized

by a feature vector 𝑤𝑗

• The edge between the 𝑗𝑢ℎ and

𝑘𝑢ℎ nodes carries a weight 𝑥𝑗𝑘 ∝ 𝑒 𝑤𝑗, 𝑤𝑘

−1

• The degree matrix:
• Graph Laplacian:

𝑤1 𝑤2 𝑤3 𝑤4 𝑤5 𝑤6 𝑤8 𝑤7 𝑤9 𝑤12 𝑤11 𝑤10 𝑤13

6

Basic Notations

Dictionary Learning for Graph Signals By: Yael Yankelevsky

𝑔

1

𝑔

2

𝑔

3

𝑔

4

𝑔

5

𝑔

6

𝑔

8

𝑔

7

𝑔

9

𝑔

12

𝑔

11

𝑔

10

𝑔

13

• The 𝑗𝑢ℎ node has a value 𝑔

𝑗

f = graph signal

• The combinatorial Laplacian is a

differential operator:

• Defines global regularity on the

graph (Dirichlet energy): 𝑤1 𝑤2 𝑤3 𝑤4 𝑤5 𝑤6 𝑤8 𝑤7 𝑤9 𝑤12 𝑤11 𝑤10 𝑤13

7

Basic Notations

Dictionary Learning for Graph Signals By: Yael Yankelevsky

 Ignore structure (MOD, K-SVD) [Engan et al. ’99],[Aharon et al. ’06]  Analytic transforms

• Graph Fourier transform [Sandryhaila & Moura ’13]
• Windowed Graph Fourier transform [Shuman et al. ’12]
• Graph Wavelets [Coifman & Maggioni ’06],[Gavish et al. ’10],[Hammond

et al. ’11],[Ram et al. ’12],[Shuman et al. ’16],…

 Structured learned dictionaries [Zhang et al. ’12],[Thanou et al. ’14]

Related Work: Dictionaries for Graph Signals

8

Our solution: Graph Regularized Dictionary Learning

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Construct 2 graphs capturing the feature dependencies and the data manifold structure

The Basic Concept

9

Lc L Y

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Example: Traffic Dataset

10

Y

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Introduce graph regularization terms that preserve these structures

Imposed smoothness (graph Dirichlet energy):

Dual Graph Regularized Dictionary Learning

11

Dictionary Learning for Graph Signals By: Yael Yankelevsky

A good estimation of L is crucial! We can learn L and adapt it to promote the desired smoothness [Hu et al. ’13], [Dong et al. ‘15],

[Kalofolias ’16], [Segarra et al. ‘17],…

12

The Importance of the Underlying Graph

[Shuman et al. ’13]

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Dual Graph Regularized Dictionary Learning

13

Key idea: Dictionary atoms preserve feature similarities Similar signals have similar sparse representations The graph is adapted to promote the desired smoothness

Dictionary Learning for Graph Signals By: Yael Yankelevsky

The DGRDL Algorithm

14

Initialize Dictionary Sparse Coding

Using ADMM pursuit

Dictionary Update

Atom-by-atom + coeffs. (modified update rule)

Graph Update

Dictionary Learning for Graph Signals By: Yael Yankelevsky

The DGRDL Algorithm

15

Initialize Dictionary Sparse Coding

Using ADMM pursuit

Dictionary Update

Atom-by-atom + coeffs. (modified update rule)

Graph Update For the j-th atom:

?

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Graph Regularized Pursuit

16

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Graph Regularized Pursuit

17

ADMM: [Boyd et al. ’11]

Dictionary Learning for Graph Signals By: Yael Yankelevsky

ADMM: [Boyd et al. ’11]

Graph Regularized Pursuit

18

graph SC [Zheng et al. ’11]

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Classical sparse theory:

Theoretical Guarantees

19

Theorem: If the true representation 𝐲 satisfies 𝐲 0 = s < 1 2 1 + 1 μ 𝐄 then a solution 𝐲 for (𝐐0

ϵ) must be close to it

𝐲 − 𝐲 2

2 ≤

4ϵ2 1 − δ2s ≤ 4ϵ2 1 − 2s − 1 μ 𝐄

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Graph sparse coding:

Theoretical Guarantees

20

Theorem: If the true representation 𝐘 satisfies 𝐘 0,∞ = s < 1 2 1 + 1 + f β, 𝐌𝐝 μ 𝐄 then a solution 𝐘 for (𝐐𝟏,∞

ϵ

) must be close to it 𝐘 − 𝐘 F

2 ≤

4ϵ2 1 − δ2s ≤ 4ϵ2 1 − 2s − 1 μ 𝐄 + f β, 𝐌𝐝

≥ 0

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Back to DGRDL…

21

Initialize Dictionary Sparse Code

Using ADMM pursuit

Dictionary Update

Atom-by-atom + coeffs. (modified update rule)

dictionary atoms are smooth graph signals similar signals have similar sparse codes graph is adapted to promote smoothness

Graph Update

Dictionary Learning for Graph Signals By: Yael Yankelevsky

22

Results: Network Data Recovery

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Settings:

 N=578 sensors  M=2892 signals

• 1500 for training
• 1392 for testing

 Graph signal = daily avg.

bottleneck (min.) measured at each station

Traffic Dataset

23

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Representation

24

. . .

Original signals

Sparse Coding

. . .

Reconstructed signals

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Representation

25

K-SVD GRDL (fixed L) GRDL (learned L) DGRDL

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Denoising

26

. . .

Original signals

Sparse Coding

+

AWGN

. . .

Noisy signals

. . .

Reconstructed signals

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Denoising

27

K-SVD GRDL (fixed L) GRDL (learned L) DGRDL

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Inpainting

28

. . .

Original signals

Sparse Coding

Incomplete signals

Discard p% of the samples randomly

. . . . . .

Reconstructed signals

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Inpainting

29

K-SVD GRDL (fixed L) GRDL (learned L) DGRDL

Dictionary Learning for Graph Signals By: Yael Yankelevsky

30

Results:

Image Denoising Revisited

Dictionary Learning for Graph Signals By: Yael Yankelevsky

𝑧𝑘 𝑧𝑗

𝑁 𝑜2

A Glimpse at Image Processing

31

𝑜 𝑜

• 𝐌 is an 𝑜 × 𝑜 grid (patch structure)
• 𝐄 is learned from only 1000 patches

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Image Denoising (σ=25)

32 Original Noisy (20.18dB) K-SVD (28.35dB) DGRDL (28.50dB) Original Noisy (20.18dB) K-SVD (30.56dB) DGRDL (30.71dB)

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Learn the underlying patch structure (pixel dependencies) from the data

Structure Inference

33

input image learned L

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Time to Conclude…

34

We developed an efficient algorithm for joint learning of the dictionary and the graph We have shown how sparsity-based models become applicable also for graph structured data We demonstrated how various applications can benefit from the new model Processing data is enabled by an appropriate modeling that exposes its inner structure Extensions include supervised dictionary learning and supporting high dimensions

Dictionary Learning for Graph Signals By: Yael Yankelevsky

35

Thank You