SLIDE 1
1/ 9
Fast greedy algorithms for dictionary selection with generalized sparsity constraints
Kaito Fujii & Tasuku Soma (UTokyo)
Neural Information Processing Systems 2018, spotlight presentation
- Dec. 7, 2018
Fast greedy algorithms for dictionary selection with generalized - - PowerPoint PPT Presentation
Fast greedy algorithms for dictionary selection with generalized - - PowerPoint PPT Presentation
Fast greedy algorithms for dictionary selection with generalized sparsity constraints Kaito Fujii & Tasuku Soma (UTokyo) Neural I nformation Processing Systems 2018, spotlight presentation Dec. 7, 2018 1/ 9 Dictionary I f real-world signals
SLIDE 2
SLIDE 3
2/ 9
Dictionary
If real-world signals consist of a few patterns, a “good” dictionary gives sparse representations of each signal
patch
SLIDE 4
2/ 9
Dictionary
If real-world signals consist of a few patterns, a “good” dictionary gives sparse representations of each signal
patch dictionary
SLIDE 5
2/ 9
Dictionary
If real-world signals consist of a few patterns, a “good” dictionary gives sparse representations of each signal
patch dictionary sparse representation
≈ ≈ ≈ 0.4 +0.1 +0.9 0.2 +0.2 +0.3 0.5 +0.5 +0.1
SLIDE 6
3/ 9
Dictionary selection [Krause–Cevher’10]
Union of existing dictionaries DCT basis Haar basis Db4 basis Coiflet basis Selected atoms as a dictionary Atoms for each patch yt (∀t ∈ [T])
SLIDE 7
3/ 9
Dictionary selection [Krause–Cevher’10]
Union of existing dictionaries DCT basis Haar basis Db4 basis Coiflet basis Selected atoms as a dictionary Atoms for each patch yt (∀t ∈ [T])
SLIDE 8
3/ 9
Dictionary selection [Krause–Cevher’10]
Union of existing dictionaries DCT basis Haar basis Db4 basis Coiflet basis Selected atoms as a dictionary ≈ w1 +w2 +w3 ≈ w1 +w2 +w3 Atoms for each patch yt (∀t ∈ [T])
SLIDE 9
3/ 9
Dictionary selection [Krause–Cevher’10]
Union of existing dictionaries DCT basis Haar basis Db4 basis Coiflet basis Selected atoms as a dictionary ≈ w1 +w2 +w3 ≈ w1 +w2 +w3 Atoms for each patch yt (∀t ∈ [T])
X Z1 Z2
SLIDE 10
4/ 9
Dictionary selection with sparsity constraints
Maximize
X⊆V
max
(Z1,··· ,ZT)∈I : Zt⊆X T
∑
t=1
ft(Zt) subject to |X| ≤ k 1st maximization: selecting a set X of atoms as a dictionary
SLIDE 11
4/ 9
Dictionary selection with sparsity constraints
Maximize
X⊆V
max
(Z1,··· ,ZT)∈I : Zt⊆X T
∑
t=1
ft(Zt) subject to |X| ≤ k 2nd maximization: selecting a set Zt ⊆ X of atoms for a sparse representation of each patch under sparsity constraint I
SLIDE 12
4/ 9
Dictionary selection with sparsity constraints
Maximize
X⊆V
max
(Z1,··· ,ZT)∈I : Zt⊆X T
∑
t=1
ft(Zt) subject to |X| ≤ k
set function representing the quality of Zt for patch yt sparsity constraint
SLIDE 13
4/ 9
Dictionary selection with sparsity constraints
Maximize
X⊆V
max
(Z1,··· ,ZT)∈I : Zt⊆X T
∑
t=1
ft(Zt) subject to |X| ≤ k
set function representing the quality of Zt for patch yt sparsity constraint
Our contributions Our contributions 1 Replacement OMP: A fast greedy algorithm with approximation ratio guarantees
SLIDE 14
4/ 9
Dictionary selection with sparsity constraints
Maximize
X⊆V
max
(Z1,··· ,ZT)∈I : Zt⊆X T
∑
t=1
ft(Zt) subject to |X| ≤ k
set function representing the quality of Zt for patch yt sparsity constraint
Our contributions Our contributions 1 Replacement OMP: A fast greedy algorithm with approximation ratio guarantees 2 p-Replacement sparsity families: A novel class of sparsity constraints generalizing existing ones
SLIDE 15
5/ 9
1 Replacement OMP
Replacement Greedy for two-stage submodular maximization [Stan+’17]
SLIDE 16
5/ 9
1 Replacement OMP
Replacement Greedy for two-stage submodular maximization [Stan+’17] Replacement Greedy O(s2dknT) running time 1st result application to dictionary selection
SLIDE 17
5/ 9
1 Replacement OMP
Replacement Greedy for two-stage submodular maximization [Stan+’17] Replacement Greedy O(s2dknT) running time 1st result application to dictionary selection Replacement OMP O((n + ds)kT) running time 2nd result O(s2d) acceleration with the concept of OMP
SLIDE 18
6/ 9
1 Replacement OMP
algorithm approximation ratio running time empirical performance SDSMA [Krause–Cevher’10] SDSOMP [Krause–Cevher’10] Replacement Greedy Replacement OMP
SLIDE 19
7/ 9
2 p-Replacement sparsity families
individual sparsity [Krause–Cevher’10] individual matroids [Stan+’17] block sparsity [Krause–Cevher’10] average sparsity w/o individual sparsity average sparsity [Cevher–Krause’11]
⊆ ⊆ ⊆ ⊆
SLIDE 20
7/ 9
2 p-Replacement sparsity families
individual sparsity [Krause–Cevher’10] individual matroids [Stan+’17] block sparsity [Krause–Cevher’10] average sparsity w/o individual sparsity average sparsity [Cevher–Krause’11]
⊆ ⊆ ⊆ ⊆ k-replacement sparse (2k − 1)-replacement sparse (3k − 1)-replacement sparse
SLIDE 21
8/ 9
2 p-Replacement sparsity families
We extend Replacement OMP to p-replacement sparsity families Theorem Replacement OMP achieves
m2
2s
M2
s,2
- 1 − exp
- − k
p Ms,2 m2s
- approximation
if I is p-replacement sparse
Assumption ft(Zt)
△
= max
wt : supp(wt)⊆Zt
ut(wt) where ut is m2s-strongly concave on Ω2s = {(x, y): ∥x − y∥0 ≤ 2s} and Ms,2-smooth on Ωs,2 = {(x, y): ∥x∥0 ≤ s, ∥y∥0 ≤ s, ∥x − y∥0 ≤ 2}
SLIDE 22