Learning sparsely used overcomplete dictionaries Alekh Agarwal - - PowerPoint PPT Presentation

Learning sparsely used overcomplete dictionaries

Alekh Agarwal Microsoft Research

Joint work with Anima Anandkumar, Prateek Jain, Praneeth Netrapalli and Rashish Tandon

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Motivation I: Feature learning

Practice

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Motivation I: Feature learning

Practice Papers Features

1.2 0.8 1.5 3.5 2 0.1 0.3 0.7 0.8

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Motivation I: Feature learning

Practice Papers

Feature eng.

Features

1.2 0.8 1.5 3.5 2 0.1 0.3 0.7 0.8

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Motivation I: Feature learning

Practice Papers

Feature eng.

Features

1.2 0.8 1.5 3.5 2 0.1 0.3 0.7 0.8

Feature engineering takes considerable time and skill Typically critical to good performance

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Motivation I: Feature learning

Practice Papers

Feature eng.

Features

1.2 0.8 1.5 3.5 2 0.1 0.3 0.7 0.8

Feature engineering takes considerable time and skill Typically critical to good performance Can we learn good features from data?

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Motivation II: Signal compression

Expensive to store high-dimensional signals

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Motivation II: Signal compression

Expensive to store high-dimensional signals Sparse signals have compact representation

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Motivation II: Signal compression

Expensive to store high-dimensional signals Sparse signals have compact representation Can we learn a representation where signals of interest are sparse?

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Dictionary learning in practice

Image compression (Bruckstein et al., 2009)

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Dictionary learning in practice

Image compression (Bruckstein et al., 2009) Similar successes in image denoising, inpainting, superresolution, . . .

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Dictionary learning in practice

Image compression (Bruckstein et al., 2009) Similar successes in image denoising, inpainting, superresolution, . . . Non-convex optimization, limited theoretical understanding

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Dictionary learning setup

Goal Find a dictionary with r elements such that each data point is a combination of only s dictionary elements.

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Dictionary learning setup

Goal Find a dictionary with r elements such that each data point is a combination of only s dictionary elements.

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Dictionary learning setup

Goal Find a dictionary with r elements such that each data point is a combination of only s dictionary elements.

Examples Coefficients Dictionary

= Y A∗ X ∗

Encode faces using dictionary rather than pixel values Sparsity for compression, signal processing . . .

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Dictionary learning setup

Goal Find a dictionary with r elements such that each data point is a combination of only s dictionary elements.

Examples Dictionary Coefficients

d

=

n r n r

X ∗ Y A∗

d

Topic models, overlapping clustering, image representation Overcomplete setting, r ≫ d relevant in practice

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Alternating minimization

Objective min

A,X X1

• i,j |Xij|

subject to Y = AX Dominant approach in practice Start with initial dictionary A(0) Sparse regression for coefficients given dictionary X(t + 1)i = arg min

x∈Rr x1 s.t.

Yi − A(t)x2 ≤ ǫt

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Alternating minimization

Objective min

A,X X1

• i,j |Xij|

subject to Y = AX Dominant approach in practice Start with initial dictionary A(0) Sparse regression for coefficients given dictionary Least squares for dictionary given coefficients A(t + 1) = YX(t + 1)+ i.e. Y ≈ A(t + 1)X(t + 1)

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Alternating minimization

Objective min

A,X X1

• i,j |Xij|

subject to Y = AX Dominant approach in practice Start with initial dictionary A(0) Sparse regression for coefficients given dictionary Least squares for dictionary given coefficients A(t + 1) = YX(t + 1)+ i.e. Y ≈ A(t + 1)X(t + 1) Similar to EM for this problem

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Alternating minimization

Objective min

A,X X1

• i,j |Xij|

subject to Y = AX Dominant approach in practice Start with initial dictionary A(0) Sparse regression for coefficients given dictionary Least squares for dictionary given coefficients A(t + 1) = YX(t + 1)+ i.e. Y ≈ A(t + 1)X(t + 1) Similar to EM for this problem Does not converge to global optimum from arbitrary A(0)

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Alternating minimization goal

( A, X) = min

A,X X1

subject to Y = AX Y = AX is a non-convex constraint Average of solutions is not a solution! Y = AX, Y = (−A)(−X),

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Alternating minimization goal

( A, X) = min

A,X X1

subject to Y = AX Y = AX is a non-convex constraint Average of solutions is not a solution! Y = AX, Y = (−A)(−X), Y = A + (−A) 2 X + (−X) 2

• Non-convex optimization, NP-hard in general

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Previous theory work

Exact recovery in undercomplete setting by Spielman et al. via linear programming

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Previous theory work

Exact recovery in undercomplete setting by Spielman et al. via linear programming We combine alternating minimization with a novel initialization Global optimum despite non-convexity in overcomplete setting

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Initialization: Key ideas

Find several samples with a common dictionary element

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Initialization: Key ideas

Find several samples with a common dictionary element Top singular vector of these samples is an estimate of this element

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Correlation graph

Definition (Correlation graph) One node for each example Large correlation ⇒ common dictionary element

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Correlation graph

Definition (Correlation graph) One node for each example Edge {Yi, Yj} if |Yi, Yj| ≥ ρ Large correlation ⇒ common dictionary element

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Correlation graph

Definition (Correlation graph) One node for each example Edge {Yi, Yj} if |Yi, Yj| ≥ ρ Large correlation ⇒ common dictionary element

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Correlation graph

Definition (Correlation graph) One node for each example Edge {Yi, Yj} if |Yi, Yj| ≥ ρ

Bad S1 S2 Good

Large correlation ⇒ common dictionary element Samples in a clique contain a common dictionary element

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Correlation graph

Definition (Correlation graph) One node for each example Edge {Yi, Yj} if |Yi, Yj| ≥ ρ

Bad Good S1 S2

Large correlation ⇒ common dictionary element Samples in a clique contain a common dictionary element Easy to construct cliques

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Correlation graph

Definition (Correlation graph) One node for each example Edge {Yi, Yj} if |Yi, Yj| ≥ ρ

Good Bad S1 S2

Large correlation ⇒ common dictionary element Samples in a clique contain a common dictionary element Easy to construct cliques

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Initialization algorithm

• 1. Construct correlation graph Gρ given a threshold ρ
• 2. For each edge (Yi, Yj) in Gρ

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Initialization algorithm

• 1. Construct correlation graph Gρ given a threshold ρ
• 2. For each edge (Yi, Yj) in Gρ

If (Yi, Yj) is good

(a) Let S be all common neighbors of Yi and Yj

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Initialization algorithm

• 1. Construct correlation graph Gρ given a threshold ρ
• 2. For each edge (Yi, Yj) in Gρ

If (Yi, Yj) is good

(a) Let S be all common neighbors of Yi and Yj (b) Let M be the covariance matrix of S:

i∈S YiY T i

(c) Set a to the top singular vector of M

• 3. Each vector

a is estimate of some A∗

i

Similar algorithm developed simultaneously and independently in Arora et al. (2013)

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Assumptions

Incoherent dictionary: |A∗

i , A∗ j | ≤ µ0/

√ d

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Assumptions

Incoherent dictionary: |A∗

i , A∗ j | ≤ µ0/

√ d Sparse coefficients: Each sample has at most s non-zero X ∗

ij with

random sparsity pattern

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Exact recovery

Theorem (AAJNT’13) Suppose we have O(r2) examples. Use graph clustering algorithm to initialize alternating minimization. With high probability, for all t ≥ 1 and i = 1, 2, . . . , r A(t)i − A∗

i 2 ≤ A(0)i − A∗ i 2 2−t

Exact recovery from O(r2) samples Global optimum through novel initialization

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Exact recovery

Theorem (AAJNT’13) Suppose we have O(r2) examples. Use graph clustering algorithm to initialize alternating minimization. With high probability, for all t ≥ 1 and i = 1, 2, . . . , r A(t)i − A∗

i 2 ≤ A(0)i − A∗ i 2 2−t

Exact recovery from O(r2) samples Global optimum through novel initialization Approximate recovery in initialization step Local linear convergence of alternating minimization

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Local linear convergence

1 2 3 4 5 6 10

−6

10

−4

10

−2

10 Error vs Iteration (d=100, r=200, s=3) Iteration No. Error in A (log scale) n=1.5 s r log r n=2 s r log r n=2.5 s r log r n=3 s r log r n=3.5 s r log r

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

One-shot vs alternating

1.5 2 2.5 3 3.5 10

−10

10

−5

10 Error vs N (d=100, r=100, s=3, n=C s r log(r)) Error in A (log scale) C (n= C s r log(r)) Initialization Alternating Minimization

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Sample complexity

1.5 2 2.5 3 3.5 4 4.5 0.2 0.4 0.6 0.8 1 n/r

• Prob. of success

d=200,s=5 r=d r=2d r=4d r=8d

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Alternating minimization proof sketch

Ideally want

X ∗ A∗ A(0)

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Alternating minimization proof sketch

Ideally want

X ∗ A∗ A(0) X(1)

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Alternating minimization proof sketch

Ideally want

A∗ X ∗ A(0) X(1) A(1)

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Alternating minimization proof sketch

Ideally want

X(2) X ∗ A∗ A(0) X(1) A(1)

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Alternating minimization proof sketch

Ideally want

X(2) X ∗ A∗ A(0) X(1) A(1)

But what about

X(2) X ∗ A∗ A(0) X(1) A(1)

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Alternating minimization proof sketch (contd.)

Lemma Suppose X(t + 1) − X ∗∞ = O(1/s). Then A(t + 1)i − A∗

i 2 = O

s2 √ d X(t + 1) − X ∗∞

• .

s2 ≤ √ d ensures error decreases Contraction by relating X(t + 1) − X ∗∞ to A(t)i − A∗

i 2

Good initialization ensures precondition

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

Conclusions

Provable recovery of overcomplete dictionaries Global optimality through novel initialization Local linear convergence of alternating minimization Local convexity under same initialization General theory for latent variable models

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning

A Clustering Approach to Learn Sparsely-Used Overcomplete Dictionaries, arxiv:1309.1952 Learning Sparsely Used Overcomplete Dictionaries via Alternating Minimization, arxiv:1310.7991 Questions?

Agarwal, Anandkumar, Jain, Netrapalli, Tandon Overcomplete Dictionary Learning