A A Modi dified d Frank nk-Wo Wolfe Algorithm for Te Tensor Fa - - PowerPoint PPT Presentation

A A Modi dified d Frank nk-Wo Wolfe Algorithm for Te Tensor Fa Factorization with Unimodal Signals

Junting Chen

The Chinese University of Hong Kong, Shenzhen Guangdong, China

Urbashi Mitra

University of Southern California CA, USA

Acknowledgement This research has been funded in part by one or more of the following grants: ONR N00014-15-1-2550, NSF CNS-1213128, NSF CCF-1718560, NSF CCF-1410009 , NSF CPS- 1446901, and AFOSR FA9550-12-1-0215.

Ma Many signals to be es>mated have unimodal pr proper> per>es es

CUHK-Shenzhen Unimodal Frank-Wolfe 2

Estimate spectra of different chemicals from compound samples False estimate (known as

• utlier) when

not exploiting unimodality

(Bro&Sidiropoulos’98)

Known spectrum curves (unimodal) acous^c sensors in the ocean Signal propaga^on (spa^ally unimodal) Source localiza^on exploi^ng only unimodality in an unknown environment (e.g., underwater)

Fo Formula>on: Add a uni unimoda dality co constraint to im improve the es>m >ma>o >on

CUHK-Shenzhen Unimodal Frank-Wolfe 3

P : minimize

x∈Rn

f(x) subject to x ∈ U ∩ M

a 3D unimodal cone intersected a sphere cost func^on for the es^ma^on problem some other constraints. Here, we assume: M = {x : a  kxk1  b} non-convex! U an n-dimensional vector

Goal: Design low complexity algorithms

the set of all unimodal vectors

… …

s s - 1 s + 1

xs xs−1 xs+1

0 ≤ x1 ≤ x2 · · · ≤ xs xx+1 ≥ xs+2 ≥ · · · ≥ xn ≥ 0

Projec^on will be expensive when we need it for very update!

CUHK-Shenzhen Unimodal Frank-Wolfe 4

Prior work mainly focused on projec^ons:

Ø For simple objec^ves (e.g., least-squares,

L1, L-infinity norm):

§ Fast isotonic projec^on: Németh&Németh’10 Prefix isotonic regression: Stout’10 § Complexity: roughly O(n) – O(n2)

Ø For general objec^ve: use projec^on

§ Alterna^ng least-squares with unimodal projec^on: Bro&Sidiropoulos’98 § projected gradient: Chen&Mitra’17

… …

s s - 1 s + 1

xs xs−1 xs+1

x(t+1) = PU h x(t) + λtrf(x(t)) i

prefix isotonic projection

Can Can we design low complexity y projection-fr free me meth thod

• ds?

CUHK-Shenzhen Unimodal Frank-Wolfe 5

(Garber&Hazan)

x(t+1) = x(t) + λt(ˆ y − x(t)) ˆ y = arg min

y∈M rf(x(t))Ty

If the constraint set is convex, then the Frank- Wolfe update can guarantee to stay inside the constraint set. Not the case here!

The Frank-Wolfe update procedure (no projection required)

Pr Proposed design: successive ve linear approximation co could be a way to handle the non-co convex co constraint

CUHK-Shenzhen Unimodal Frank-Wolfe 6

f(x) U ∩ M

Convex local constraint set Original constraint set (non-convex) minimize

y∈Rn

f(x(t)) + rf(x(t))Ty subject to y 2 U(x(t)) x(t+1) = x(t) + λt(ˆ y − x(t))

dynamically construct a convex constraint set Frank-Wolfe update

Ne New chal allenges: need to dyn ynam amical ally y design the co convex co constraint set U(x (x(t))

CUHK-Shenzhen Unimodal Frank-Wolfe 7

f(x) U ∩ M

Convex local constraint set Original constraint set (non-convex) minimize

y∈Rn

f(x(t)) + rf(x(t))Ty subject to y 2 U(x(t)) x(t+1) = x(t) + λt(ˆ y − x(t)) Challenge 1: The sub- problems need to be solved efficiently Challenge 2: Needs to justify the convergence O(n) complexity or better

Pr Property: The union of two adjacent components ar are convex but three ar are non-co convex

CUHK-Shenzhen Unimodal Frank-Wolfe 8

U3

U4

U5

U6

U7

set of unimodal vectors with the sth element being the largest

Choice 1:

Us =

x ∈ Rn :

… …

s s - 1 s + 1 xs xs−1 xs+1

x(t)

Choice 2: U(x(t)) = U4 ∪ U5 ? U(x(t)) = U4 ∪ U5 ∪ U6 ?

U(x (x(t)) ) needs to be convex such th that t th the sub- pr probl blem ems s can n be be so solved ed effi fficien ently

CUHK-Shenzhen Unimodal Frank-Wolfe 9

U3

U4

U5

U6

U7

set of unimodal vectors with the sth element being the largest

not included in the constraint set

U ∩ M

conv( e U5)

Us =

x ∈ Rn :

x(t)

Choice 2’: U(x(t)) = conv(U4 ∪ U5 ∪ U6) ?

Th The update always stay inside th the constr traint t set, t, i. i.e., bein ing g unim imodal

CUHK-Shenzhen Unimodal Frank-Wolfe 10

U(x) := conv e US(x)

• ∩ L(x)

L(x) , {y 2 Rn : a(x)  kyk1  b(x)} conv( ˜ Us) L(x) U(x) kxk1 a(x) kxk1  b(x) a(x) = min{1, ˆ α(x)}kxk1 b(x) = max{1, ˆ α(x)}kxk1

ˆ α = argmin

α≥0,αx∈M

f(αx)

where

U3

U4

U5

U6

U7

U(x(t))

x(t)

ˆ y

x(t+1)

Property: U(x) is a convex polytope with at most 2n extreme points à one being the solution to the LP à found in 2n steps by a simplex algorithm

rf(x(t))

Wh Why the sub-pr probl blem em can n be be comput puted ed effi fficien ently?

CUHK-Shenzhen Unimodal Frank-Wolfe 11

f(x) U ∩ M Convex polytopes Original constraint set (non-convex)

… …

s s - 1 s + 1

xs xs−1 xs+1

Each U(x(t)) is a convex polytope with at most 2n extreme points a simplex method can find the

• p^mal solu^on via at most 2n

steps minimize

y∈Rn

f(x(t)) + rf(x(t))Ty subject to y 2 U(x(t))

q Theorem 1. The algorithm converges under step size . q Theorem 2. Under adaptive step size, the gap

converges to zero at a rate .

§

g(x) = 0 defines a stationary point

§

g(x) gives a lower bound the duality gap

q Theorem 3. If f is strictly convex and its critical point

satisfying for any , and in addition, , then the algorithm converges to the global

• ptimal solution

from any initial point. O(1/ √ t + 1)

Th The algorith thm m converges, and global convergence is al also possible

CUHK-Shenzhen Unimodal Frank-Wolfe 12

λt = 2/(t + 2) g(x) = max

w∈U(x) rf(x)T(w x)

rf(x?) = 0 f(x1) < f(x2) kx1 x?k < ||x2 x?k x? ∈ U ∩ M x∗ = x?

q Gaussian noise corrupted observa^on z = c + n, . q Choice of objec^ve

Ex Example of Es Es>ma>ng an n-di dimensi ensiona nal uni unimoda dal si signa nal: : the he conver ergenc ence

CUHK-Shenzhen Unimodal Frank-Wolfe 13

n ∼ N(0, σ2I) f(x) = kx zk2

2

High SNR, Theorem 3 applies, global convergence Low SNR, sigma=10, mul^ple sta^onary points

Th The recovery performa mance: enforcing uni unimoda dality enha enhanc nces es the he es es>ma>on n per performanc nce

CUHK-Shenzhen Unimodal Frank-Wolfe 14

Low SNR High SNR

Unimodal projection [Stout’10] + non- negative projection

f(x) = kx zk2

2

minimize subject to x being unimodal (and non-nega^ve)

u n i m

• d

a l i t y i g n

• r

e d

Ap Applica>on >on to

• ten

ensor

• r fact

ctor

• riza>on

>on: mu mul>mo modal data à te tensor model à uni unimoda dal struc uctur ure for ea each ch layer er à uni unimoda dal FW algorithm hm

CUHK-Shenzhen Unimodal Frank-Wolfe 15

multimodal data for estimating a source N(log N)2 ∼ M sparse observa^on at each layer signals at each layer are unimodal (peaks are assumed aligned)

X ∈ RN×N×K

tensor

Th The domi minant t vectors fr from m least-squa squares es rank nk-1 1 te tensor approxima>on are unimodal

CUHK-Shenzhen Unimodal Frank-Wolfe 16

q Theorem (Chen&Mitra’18). The optimal solutions w1 and w2 of P0

are unimodal, with their peak locations correspond to the source location.

§ §

denotes the mode-p multiplication P0 : minimize

w1,w2,w3,α

kX α ⇥1 w1 ⇥2 w2 ⇥3 w3k2

F

subject to α > 0, kw1k = kw2k = kw3k = 1 kXk2

F , P i

P

j

P

k X(i, j, k)2

X ×p A

w1 w2

w1 w2 w3 ≈ α×

(full observa^on)

Te Tensor factoriza>on enforcing uni unimoda dality co constraints

CUHK-Shenzhen Unimodal Frank-Wolfe 17

Proposed modified Frank-Wolfe algorithm applies to update w1, w2, and w3 alternatively using the gradients:

PUTF : minimize

α,w1,w2,w3

kW ✓ (X α ⇥1 w1 ⇥2 w2 ⇥3 w3)k2

F

subject to α > 0, kw1k1 = kw2k1 = kw3k1 = 1. w1, w2 2 U

1 2 ∂f ∂w1 = −a(Xw

(1))T(w3 ✏ w2) + a2⇥

W T

(1)(w2 3 ✏ w2 2)

⇤ ✓ w1 1 2 ∂f ∂w2 = −a(Xw

(2))T(w3 ✏ w1) + a2⇥

W T

(2)(w2 3 ✏ w2 1)

⇤ ✓ w2 1 2 ∂f ∂w3 = −a(Xw

(3))T(w2 ✏ w1) + a2⇥

W T

(3)(w2 2 ✏ w2 1)

⇤ ✓ w3

Loc Localization

• n: the

e same e sou

• urce

e em emitting two

• typ

ypes es of

• f

si signal als s being cap aptu tured by RSS SS an and TOA se senso sors

CUHK-Shenzhen Unimodal Frank-Wolfe 18

q How data is generated:

50% for RSS of the EM signal 50% for TOA of the acous^c signal

q Preprocessing: Data normaliza^on

Betas are chosen such that the normalized data is roughly uniform; N is the largest number sa^sfying

PdB(d) = 70 − 36 × log10(max{10, d}) + N(0, σ2

s )

σs = 10 dB t(d) = d 340 m/s + N(0, σ2

t )

σt = 100 ms h1(d) = exp(−β110−PdB(d)/10) h2(d) = exp(−β2t(d)2) 1.5N(log N)2 ≤ P

k |Mk|

En Enforci cing uni unimoda dality in indeed im improves the es es>ma>on

CUHK-Shenzhen Unimodal Frank-Wolfe 19

tensor-based methods Matrix-based methods

Unimodaityl-non-aware (weighted centroid)

ˆ sRSS = P

m∈RRSS %(m)z(m)

P

m∈RRSS %(m)

Enforcing unimodality

Te Tensor factoriza>on strategy fuses mul>modal data be beTer er

CUHK-Shenzhen Unimodal Frank-Wolfe 20

When one of the signal modes (TOA signal) is corrupted… Tensor-based methods: robust to the deteriora^on

• f the signals

Matrix-based methods: sensi^ve to the deteriora^on

• f the signals

Su Substan>al comp

• mplexity

ty reduc>on

• n by th

the prop

• pos
• sed

uni unimoda dal-FW FW algorithm hm

CUHK-Shenzhen Unimodal Frank-Wolfe 21

Projected gradient based on unimodal regression ([Stout’08], state-of-the-art) Proposed Unimodal Frank-Wolfe

Substan^al complexity reduc^on

In In c conclusion, w we d developed a a u unimodal-FW FW al algorith thm to so solve un unimodal ality-co constrained problems

CUHK-Shenzhen Unimodal Frank-Wolfe 22

f(x) U ∩ M

convex polytopes U(x(t)) (non-convex) minimize

x∈Rn

f(x) subject to x ∈ U ∩ M

• main idea: construct a sequence
• f linear sub-problems, each

constrained by a convex polytope

• complexity 2n for each sub-

problem

• shown to converge

(global convergence possible)

• Demonstration for a data fusion

problem using tensor model

Thank you & Ques^ons?

Bi Bibliogr graphy

CUHK-Shenzhen Unimodal Frank-Wolfe 23

q

[Bro&Sidiropoulous’98] R. Bro and N. D. Sidiropoulos, “Least squares algorithms under unimodality and non-nega^vity constraints,” Journal of Chemometrics, vol. 12, no. 4, pp. 223–247, 1998.

q

[Stout’08] Q. F. Stout, “Unimodal regression via prefix isotonic regression,” Computa;onal Sta;s;cs & Data Analysis, vol. 53, no. 2, pp. 289–297, 2008.

q

[Németh&Németh’10] A. Németh and S. Németh, “How to project onto an isotone projec^on cone,” Linear Algebra and its Applica;ons, vol. 433, no. 1, pp. 41–51, 2010.

q

[Gunn&Dunson’05] L. H. Gunn and D. B. Dunson, “A transforma^on approach for incorpora^ng monotone or unimodal constraints,” Biosta;s;cs, vol. 6, no. 3, pp. 434–449, 2005.

q

[Köllman et.al.’14] C. Köllmann, B. Bornkamp, and K. Ickstadt, “Unimodal regression using bernstein–schoenberg splines and penal^es,” Biometrics, vol. 70, no. 4, pp. 783–793, 2014.

q

[Chen&Mitra’17] J. Chen and U. Mitra, “Unimodality-Constrained Matrix Factoriza^on for Non-Parametric Source Localiza^on,” 2017, submiued to IEEE Trans. Signal Process., preprint arXiv:1711.07457.

q

[Chen&Mitra’18] J. Chen and U. Mitra, “A tensor decomposi^on technique for source localiza^on from mul^modal data,” in Proc. IEEE Int. Conf. Acous;cs, Speech, and Signal Processing, Calgary, Alberta, Canada, Apr. 2018.