Faster and still safe: Combining screening techniques and - - PowerPoint PPT Presentation

Faster and still safe: Combining screening techniques and structured dictionaries to accelerate the Lasso

03/04/18

• 1

Cássio F. DANTAS, Rémi GRIBONVAL cassio.fraga-dantas@inria.fr, remi.gribonval@inria.fr

Accelerate the Lasso optimization by combining two strategies : 1) Safe Screening Rules 2) Fast Structured Dictionaries

Contents

03/04/18

• 3

01. Context (Lasso problem) 02. Fast Structured Dictionaries 03. Screening Rules 04. Screening Rules w/ Approx. Dictionaries 05. Results 06. Conclusion

01

4

Context

01 Context

5

Lasso problem

The l1-regularized least squares. Denoting :

• the observation vector;
• the design matrix (or dictionary);
• the sparse representation vector;
• parameter controlling the sparsity of the solution.

6

Dual Lasso

Dual formulation of the Lasso problem : 01 Context Denoting :

• the dual variable;
• the feasible set;

02

7

Fast Structured Dictionaries

Fast Structured Dictionaries

8

Motivation

• Iterative algorithms are often used to solve the Lasso problem.
• Exemple : ISTA (Iterative Shrinkage-Thresholding Algorithm)
• T

wo matrix-vector multiplications at each iteration. Quadratic complexity! Can it be reduced ?

Fast Approximate Dictionaries

9

Structure ⇒ Acceleration

Accelerate matrix-vector multiplications

Constrain the dictionary matrix to have a certain type of structure. Examples :

–

Kronecker product

–

Sparse factors

–

Circulant factors

–

(...)

Fast Approximate Dictionaries

10

Structured Approximation

If the dictionary matrix is not structured, an structured approximation can be found. where is the approximation error matrix and is its -th column.

01 Context

11

Algorithm (high level view)

1) Start Lasso optimization by using the structured , to take advantage

• f

its reduced multiplication cost. 2) As the algorithm approaches the solution, switch back to the

• riginal dictionary .

01 Context

11

Algorithm (high level view)

1) Start Lasso optimization by using the structured , to take advantage

• f

its reduced multiplication cost. 2) As the algorithm approaches the solution, switch back to the

• riginal dictionary .

02

12

Safe Screening Rules

Safe Screening Rules

13

Safe Screening

• Rules for identifying inactive dictionary atoms, before completely solving the problem.

Safe Screening Rules

13

Safe Screening

• Rules for identifying inactive dictionary atoms, before completely solving the problem.

Dictionary columns that will receive zero weight

• n the reconstruction of the input signal

Safe Screening Rules

13

Safe Screening

• Rules for identifying inactive dictionary atoms, before completely solving the problem.

Dictionary columns that will receive zero weight

• n the reconstruction of the input signal

Safe Screening Rules

13

Safe Screening

• Rules for identifying inactive dictionary atoms, before completely solving the problem.

Dictionary columns that will receive zero weight

• n the reconstruction of the input signal

Inactive atoms Solution support

Safe Screening Rules

13

Safe Screening

• Rules for identifying inactive dictionary atoms, before completely solving the problem.
• We can eliminate such atoms.
• Zero risk of false eliminations !

Dictionary columns that will receive zero weight

• n the reconstruction of the input signal

Inactive atoms

Screening T est

• Function
• f the atom

is surely inactive.

14

Screening T est

• Function
• f the atom

is surely inactive. Rejection set: Preserved set:

14

Safe Screening Rules

15

Screening T est - Details

Given a region (safe region) which contains .

Sphere test

Safe region is a closed l2-ball with center c and radius r :

04

16

Screening Rules with Approximate Dictionaries

Extending Screening Rules

17

Rejection set Guarantee:

Extending Screening Rules

17

Rejection set Guarantee:

Extending Screening Rules

Rejection set Guarantee: Rejection set

17

Extending Screening Rules

Rejection set Guarantee: Rejection set

17

Extending Screening Rules

Rejection set Guarantee: Rejection set

17

Extending Screening Rules

18

Extending sphere tests

Sphere test :

A certain « security margin » must be added to account for the atom approximation error. Suppose a safe sphere given.

Extending Screening Rules

18

Sphere test with approximate dictionary :

A certain « security margin » must be added to account for the atom approximation error.

Extending sphere tests

Suppose a safe sphere given.

Sphere test :

Extending Screening Rules

19

Obtaining a safe sphere

Given a primal-dual estimation at iteration .

GAP safe sphere :

with the duality gap at iteration .

Extending Screening Rules

19

Obtaining a safe sphere

with the duality gap at iteration . Given a primal-dual estimation at iteration .

GAP safe sphere :

Extending Screening Rules

19

Obtaining a safe sphere

with the duality gap at iteration . Given a primal-dual estimation at iteration .

GAP safe sphere :

Extending Screening Rules

19

Obtaining a safe sphere

with the duality gap at iteration . Given a primal-dual estimation at iteration .

GAP safe sphere :

Extending Screening Rules

Obtaining a safe sphere

Given a primal-dual estimation at iteration .

GAP safe sphere: GAP safe sphere with approximate dictionary:

cannot be calculated, since depends on . Instead, we use a modifjed primal with

20

Dynamic screening

Rejection set Safe region GAP sphere Guarantee 1: Guarantee 2:

21

Extended dynamic screening

Safe region Extended GAP sphere Guarantee 1: Rejection set Guarantee 2:

21

We now have safe screening rules that manipule an approximate dictionary. But, what’s the impact of the numerous security margins? Is it still worth it?

05

22

Results

Results

23

Running times per iteration

• Less inactive atoms are identifjed by the extended screening.
• BUT, structured dictionary makes the initial iterations much faster.

Results

23

Running times per iteration

• Less inactive atoms are identifjed by the extended screening.
• BUT, structured dictionary makes the initial iterations much faster.

Results

23

Running times per iteration

Results

24

06

25

Conclusion

26

Conclusion

• The proposed approach combines screening rules and fast approximate dictionaries.
• It reduces even further the execution time with respect screening rules alone.

Potential extensions

• Other region types (e.g. domes)
• Other problems than Lasso (e.g. Group-Lasso, Elastic-Net, Regularized Logistic Regression)

Thank you!

Questions?

Contact me: cassio.fraga-dantas@inria.fr

Safe Screening Rules

11

Screening test – Details

Dual formulation of the Lasso problem : Projection problem ! At the dual solution : ➢ Constraints on and are active, i.e. ➢ Constraints on is inactive, i.e.

Feasible region

Safe Screening Rules

12

Screening test – Details

• Every dictionary atom for which

is inactive.

• Then, simply calculate for all and discard all atoms for which the result is

smaller than 1. Dual solution is not known. Identify a region (safe region) which contains . Suffjcient condition : is inactive

Extending Screening Rules

Swithing criterion

Reasons to switch back from to :

• Convergence: to avoid converging to the solution of the approximate problem.
• The higher the approximation error, the sooner we need to switch.
• Screening ratio: the number of active atoms may become so small that the use of

does not pay ofg anymore.

Extending Screening Rules

23

Comparison

Less inactive atoms are identifjed by the extended screening.

Extending Screening Rules

24

Swithing criterion

Extending Screening Rules

25

Complexity reduction

Extending Screening Rules

25

Complexity reduction

Simulation Results

27

Impact of the Approximation Error