Fast greedy algorithms for dictionary selection with generalized - PowerPoint PPT Presentation

Fast greedy algorithms for dictionary selection with generalized sparsity constraints Kaito Fujii & Tasuku Soma (UTokyo) Neural I nformation Processing Systems 2018, spotlight presentation Dec. 7, 2018 1/ 9

Dictionary I f real-world signals consist of a few patterns, a “ good ” dictionary gives sparse representations of each signal 2/ 9

Dictionary I f real-world signals consist of a few patterns, a “ good ” dictionary gives sparse representations of each signal patch 2/ 9

Dictionary I f real-world signals consist of a few patterns, a “ good ” dictionary gives sparse representations of each signal patch dictionary 2/ 9

Dictionary I f real-world signals consist of a few patterns, a “ good ” dictionary gives sparse representations of each signal patch sparse representation dictionary ≈ + 0 . 1 + 0 . 9 0 . 4 ≈ + 0 . 2 + 0 . 3 0 . 2 ≈ + 0 . 5 + 0 . 1 0 . 5 2/ 9

Dictionary selection [Krause – Cevher ’ 10] Union of existing dictionaries DCT basis Haar basis Db4 basis Coi fl et basis Selected atoms as a dictionary Atoms for each patch y t ( ∀ t ∈ [ T ] ) 3/ 9

Dictionary selection [Krause – Cevher ’ 10] Union of existing dictionaries DCT basis Haar basis Db4 basis Coi fl et basis Selected atoms as a dictionary Atoms for each patch y t ( ∀ t ∈ [ T ] ) 3/ 9

Dictionary selection [Krause – Cevher ’ 10] Union of existing dictionaries DCT basis Haar basis Db4 basis Coi fl et basis Selected atoms as a dictionary Atoms for each patch y t ( ∀ t ∈ [ T ] ) ≈ w 1 ≈ w 1 + w 2 + w 3 + w 2 + w 3 3/ 9

Dictionary selection [Krause – Cevher ’ 10] Union of existing dictionaries DCT basis Haar basis Db4 basis Coi fl et basis Selected atoms as a dictionary X Atoms for each patch y t ( ∀ t ∈ [ T ] ) ≈ w 1 ≈ w 1 + w 2 + w 3 + w 2 + w 3 Z 1 Z 2 3/ 9

Dictionary selection with sparsity constraints T ∑ subject to | X | ≤ k f t ( Z t ) Maximize max X ⊆ V ( Z 1 , ··· , Z T ) ∈ I : Z t ⊆ X t = 1 1st maximization : selecting a set X of atoms as a dictionary 4/ 9

Dictionary selection with sparsity constraints T ∑ subject to | X | ≤ k f t ( Z t ) Maximize max X ⊆ V ( Z 1 , ··· , Z T ) ∈ I : Z t ⊆ X t = 1 2nd maximization : selecting a set Z t ⊆ X of atoms for a sparse representation of each patch under sparsity constraint I 4/ 9

Dictionary selection with sparsity constraints T ∑ subject to | X | ≤ k f t ( Z t ) Maximize max X ⊆ V ( Z 1 , ··· , Z T ) ∈ I : Z t ⊆ X t = 1 set function representing sparsity constraint the quality of Z t for patch y t 4/ 9

Dictionary selection with sparsity constraints T ∑ subject to | X | ≤ k f t ( Z t ) Maximize max X ⊆ V ( Z 1 , ··· , Z T ) ∈ I : Z t ⊆ X t = 1 set function representing sparsity constraint the quality of Z t for patch y t Our contributions Our contributions 1 Replacement OMP : A fast greedy algorithm with approximation ratio guarantees 4/ 9

Dictionary selection with sparsity constraints T ∑ subject to | X | ≤ k f t ( Z t ) Maximize max X ⊆ V ( Z 1 , ··· , Z T ) ∈ I : Z t ⊆ X t = 1 set function representing sparsity constraint the quality of Z t for patch y t Our contributions Our contributions 1 Replacement OMP : A fast greedy algorithm with approximation ratio guarantees 2 p -Replacement sparsity families : A novel class of sparsity constraints generalizing existing ones 4/ 9

1 Replacement OMP Replacement Greedy for two-stage submodular maximization [Stan+ ’ 17] 5/ 9

1 Replacement OMP Replacement Greedy for two-stage submodular maximization [Stan+ ’ 17] application to dictionary selection 1st result O ( s 2 dknT ) running time Replacement Greedy 5/ 9

1 Replacement OMP Replacement Greedy for two-stage submodular maximization [Stan+ ’ 17] application to dictionary selection 1st result O ( s 2 dknT ) running time Replacement Greedy O ( s 2 d ) acceleration with the concept of OMP 2nd result O (( n + ds ) kT ) running time Replacement OMP 5/ 9

1 Replacement OMP empirical approximation running algorithm performance ratio time SDS MA [Krause – Cevher ’ 10] SDS OMP [Krause – Cevher ’ 10] Replacement Greedy Replacement OMP 6/ 9

2 p -Replacement sparsity families average sparsity [Cevher – Krause ’ 11] ⊆ average sparsity w/o individual sparsity ⊆ block sparsity individual matroids ⊆ ⊆ individual sparsity [Krause – Cevher ’ 10] [Stan+ ’ 17] [Krause – Cevher ’ 10] 7/ 9

2 p -Replacement sparsity families ( 3 k − 1 ) -replacement sparse average sparsity [Cevher – Krause ’ 11] ( 2 k − 1 ) -replacement sparse ⊆ average sparsity w/o individual sparsity k -replacement sparse ⊆ block sparsity individual matroids ⊆ ⊆ individual sparsity [Krause – Cevher ’ 10] [Stan+ ’ 17] [Krause – Cevher ’ 10] 7/ 9

2 p -Replacement sparsity families We extend Replacement OMP to p -replacement sparsity families Theorem m 2 � � �� 1 − exp − k M s , 2 2 s Replacement OMP achieves -approximation M 2 p m 2 s s , 2 if I is p -replacement sparse Assumption △ f t ( Z t ) = u t ( w t ) max w t : supp ( w t ) ⊆ Z t where u t is m 2 s -strongly concave on Ω 2 s = { ( x , y ): ∥ x − y ∥ 0 ≤ 2 s } and M s , 2 -smooth on Ω s , 2 = { ( x , y ): ∥ x ∥ 0 ≤ s , ∥ y ∥ 0 ≤ s , ∥ x − y ∥ 0 ≤ 2 } 8/ 9

Overview 1 Replacement OMP : A fast algorithm for dictionary selection 2 p -Replacement sparsity families : A class of sparsity constraints Other contributions Other contributions Empirical comparison with dictionary learning methods Extensions to online dictionary selection Poster #78 at Room 210 & 230 AB, Thu 10:45 – 12:45 9/ 9