Learning ancestral atom of structured dictionary via sparse coding - - PowerPoint PPT Presentation

▶

Sep 18, 2023 258 likes •516 views

Learning ancestral atom of structured dictionary via sparse coding Bernoulli Society Satellite Meeting Noboru Murata Waseda University 2 September, 2013 joint work with Toshimitsu Aritake, Hideitsu Hino 1 / 25 sparse coding a methodology

SLIDE 1

Learning ancestral atom

f structured dictionary via sparse coding

Bernoulli Society Satellite Meeting Noboru Murata

Waseda University

2 September, 2013 joint work with Toshimitsu Aritake, Hideitsu Hino

1 / 25

SLIDE 2

sparse coding

a methodology for representing observations with a sparse combination of basis vectors (atoms) related with various problems:

associative memory (Palm, 1980) visual cortex model (Olshausen & Field, 1996) Lasso (least absolute shrinkage and selection operator; Tibshirani, 1996) compressive sensing (Cand` es & Tao, 2006) image restoration/compression (Elad et al., 2005)

2 / 25

SLIDE 3

basic problem

y = (y1, . . . , yn)T : target signal d = (d1, . . . , dn)T : atom D = (d1, . . . , dm): dictionary (redundant: m > n) x = (x1, . . . , xm)T : coefficient vector

bjective:

minimize

x

∥y − Dx∥2

2 + η∥x∥∗

where ∥ · ∥∗ is a sparse norm, e.g. ℓ0 and ℓ1.

3 / 25

SLIDE 4

dictionary design

dictionary determines overall quality of reconstruction. predefined dictionary: structured

wavelets (Daubechies, 1992) curvelets (Cand` es & Donoho, 2001) contourlets (Do & Vetterli, 2005)

learned dictionary: unstructured

gradient-based method (Olshausen & Field, 1996) Method of Optimal Directions (Engan et al., 1999) K-SVD (Aharon et al., 2006)

structured dictionary learning: intermediate

Image Signature Dictionary (Aharon & Elad, 2008) Double Sparsity (Rubinstein et al., 2010)

4 / 25

SLIDE 5

structured dictionary learning

rdered dictionary: D = (dλ; λ ∈ Λ)

typical approaches:

meta dictionary: ˜ D = ( ˜ d1, . . . , ˜ dM) dλ = ˜ D αλ where αλ is a meta-coefficient vector additional constraints are imposed on αλ, e.g. sparsity ancestral atom (ancestor): a = (a1, . . . , aN)T dλ = Fλ a where Fλ is an extraction operator

5 / 25

SLIDE 6

dictionary generation

structure: designed by a set of extraction operators extraction operator: Fp,q dp,q = Fp,q a, (p : scale or downsample level, q : shift) cut off a piece of ancestor generating operator: D D a = (dp,q; (p, q) ∈ Λ) = (Fp,q a; (p, q) ∈ Λ) a structured collection of Fp,q

6 / 25

SLIDE 7

example of extraction operators

[Fp,q]ij = { 1 j = (i − 1) × 2p + q,

therwise,

7 / 25

SLIDE 8

example of extraction operators

F0,1 =        1 · · · 1 · · · 1 · · · . . . . . . . . . ... . . . . . . · · · . . . 1 · · ·        ∈ ℜn×N F1,1 =        1 · · · 1 · · · 1 · · · . . . . . . . . . . . . . . . ... · · · . . . · · · · · ·        ∈ ℜn×N

8 / 25

SLIDE 9

basic idea

projection-based algorithm related spaces: D: space of dictionary S: space of structured dictionary A: space of ancestor related maps: dictionary learning: D → D structured dictionary: A → S ⊂ D introduce a fiber bundle structure to D by defining projection from D to S

9 / 25

SLIDE 10

ancestor aggregation

condition: G = ∑

(p,q)∈Λ

FT

p,qFp,q : A → A

is bijective where FT

p,q is the adjoint operator of Fp,q

mean operator: M : D → A M D = M (dp,q; (p, q) ∈ Λ) = G−1 ∑

p,q

FT

p,q dp,q

important relations: π = D ◦ M : D → S (projection) Id = M ◦ D : A → A (identity)

10 / 25

SLIDE 11

relation of operators

11 / 25

SLIDE 12

algorithm

procedure Ancestor Learning( a(0), D, M, U, ε > 0 ) repeat D(t) ← D a(t) ▷ generate dictionary ˜ D(t+1) ← U D(t) ▷ update dictionary a(t+1) ← M ˜ D(t+1) ▷ update ancestor until ∥a(t+1) − a(t)∥ < ε return a end procedure

12 / 25

SLIDE 13

experiment

U: OMP + K-SVD (+ renormalization)

OMP: greedy algorithm for ℓ0 norm SC K-SVD: k-means algorithm for dictionary learning

compare with ISD (Aharon & Elad, 2008)

gradient-based algorithm for estimating ancestor including only shift operation in original version

13 / 25

SLIDE 14

artificial data

ancestor (ground truth) subset of observations

14 / 25

SLIDE 15

estimated ancestors

proposed method ISD noiseless case

15 / 25

SLIDE 16

estimated ancestors

proposed method ISD noisy case

16 / 25

SLIDE 17

(a-0) examples of atom in level 0 (proposed) (c-0) spectrum of level 0 (proposed)

17 / 25

SLIDE 18

(a-0) examples of atom in level 0 (proposed) (b-0) examples of atom in level 0 (ISD) (c-0) spectrum of level 0 (proposed) (d-0) spectrum of level 0 (ISD) 18 / 25

SLIDE 19

(a-1) examples of atom in level 1 (proposed) (b-1) examples of atom in level 1 (ISD) (c-1) spectrum of level 1 (proposed) (d-1) spectrum of level 1 (ISD) 19 / 25

SLIDE 20

(a-2) examples of atom in level 2 (proposed) (b-2) examples of atom in level 2 (ISD) (c-2) spectrum of level 2 (proposed) (d-2) spectrum of level 2 (ISD) 20 / 25

SLIDE 21

images (2D atoms)

training image test image (peppers)

21 / 25

SLIDE 22

proposed method ISD estimated ancestor

22 / 25

SLIDE 23

proposed method ISD spectrum of estimated ancestor

23 / 25

SLIDE 24

proposed method ISD learned dictionary

24 / 25

SLIDE 25

concluding remarks

we proposed a dictionary generation scheme from an ancestor a condition of structured dictionary identifiabilty a projection-based algorithm to learn the ancestor possible application would be image analysis and compression frequency analysis of signals

25 / 25