[PPT] - When tensor decomposition meets compressed sensing Pierre Comon PowerPoint Presentation

SLIDE 1

Intro General Exact CP Approx CP NonNeg Coherence AAP END

When tensor decomposition meets compressed sensing

Pierre Comon

I3S, CNRS, University of Nice - Sophia Antipolis, France Collaborator: Lek-Heng Lim

Sept. 27-30, 2010

Pierre Comon LVA/ICA – Sept. 2010 1 / 43

SLIDE 2

Intro General Exact CP Approx CP NonNeg Coherence AAP END Sparse representation & BSS

Blind Source Separation & Sparse Representation

x = H s

H is K × P, underdetermined: K < P Sparse representation: Columns hn ∈ D, known dictionary BSS: H unknown

s sparse
s not sparse

Pierre Comon LVA/ICA – Sept. 2010 2 / 43

SLIDE 3

Intro General Exact CP Approx CP NonNeg Coherence AAP END Sparse representation & BSS

Blind Source Separation & Sparse Representation

x = H s

H is K × P, underdetermined: K < P Sparse representation: Columns hn ∈ D, known dictionary BSS: H unknown

s sparse
s not sparse

Pierre Comon LVA/ICA – Sept. 2010 2 / 43

SLIDE 4

Intro General Exact CP Approx CP NonNeg Coherence AAP END Sparse representation & BSS

Blind Source Separation & Sparse Representation

x = H s

H is K × P, underdetermined: K < P Sparse representation: Columns hn ∈ D, known dictionary BSS: H unknown

s sparse
s not sparse

Pierre Comon LVA/ICA – Sept. 2010 2 / 43

SLIDE 5

Intro General Exact CP Approx CP NonNeg Coherence AAP END Sparse representation & BSS

Blind Source Separation & Sparse Representation

x = H s

H is K × P, underdetermined: K < P Sparse representation: Columns hn ∈ D, known dictionary BSS: H unknown

s sparse
s not sparse

Pierre Comon LVA/ICA – Sept. 2010 2 / 43

SLIDE 6

Intro General Exact CP Approx CP NonNeg Coherence AAP END Sparse representation & BSS

Blind Source Separation & Sparse Representation

x = H s

H is K × P, underdetermined: K < P Sparse representation: Columns hn ∈ D, known dictionary BSS: H unknown

s sparse
s not sparse

Pierre Comon LVA/ICA – Sept. 2010 2 / 43

SLIDE 7

Intro General Exact CP Approx CP NonNeg Coherence AAP END Statistical approach

Blind identification of linear mixtures

Linear mixtures: x = H s If sℓ statistically independent, we have the core equation: Ψ(u) =

P

ℓ=1

ϕℓ(uTH) Take 3rd derivatives at point u: Tijk(u) =

P

ℓ=1

Hiℓ Hjℓ Hkℓ Cℓℓℓ(u) (1) At u = 0 ➽ symmetric decomposition of Tijk [HOS] At u = 0 ➽ [Taleb, Comon-Rajih, Yeredor]

Pierre Comon LVA/ICA – Sept. 2010 3 / 43

SLIDE 8

Intro General Exact CP Approx CP NonNeg Coherence AAP END Statistical approach

Blind identification of linear mixtures

Linear mixtures: x = H s If sℓ statistically independent, we have the core equation: Ψ(u) =

P

ℓ=1

ϕℓ(uTH) Take 3rd derivatives at point u: Tijk(u) =

P

ℓ=1

Hiℓ Hjℓ Hkℓ Cℓℓℓ(u) (1) At u = 0 ➽ symmetric decomposition of Tijk [HOS] At u = 0 ➽ [Taleb, Comon-Rajih, Yeredor]

Pierre Comon LVA/ICA – Sept. 2010 3 / 43

SLIDE 9

Intro General Exact CP Approx CP NonNeg Coherence AAP END Statistical approach

Blind identification of linear mixtures

Linear mixtures: x = H s If sℓ statistically independent, we have the core equation: Ψ(u) =

P

ℓ=1

ϕℓ(uTH) Take 3rd derivatives at point u: Tijk(u) =

P

ℓ=1

Hiℓ Hjℓ Hkℓ Cℓℓℓ(u) (1) At u = 0 ➽ symmetric decomposition of Tijk [HOS] At u = 0 ➽ [Taleb, Comon-Rajih, Yeredor]

Pierre Comon LVA/ICA – Sept. 2010 3 / 43

SLIDE 10

Intro General Exact CP Approx CP NonNeg Coherence AAP END Statistical approach

Blind identification of linear mixtures

Linear mixtures: x = H s If sℓ statistically independent, we have the core equation: Ψ(u) =

P

ℓ=1

ϕℓ(uTH) Take 3rd derivatives at point u: Tijk(u) =

P

ℓ=1

Hiℓ Hjℓ Hkℓ Cℓℓℓ(u) (1) At u = 0 ➽ symmetric decomposition of Tijk [HOS] At u = 0 ➽ [Taleb, Comon-Rajih, Yeredor]

Pierre Comon LVA/ICA – Sept. 2010 3 / 43

SLIDE 11

Intro General Exact CP Approx CP NonNeg Coherence AAP END Statistical approach

Blind identification of linear mixtures

Linear mixtures: x = H s If sℓ statistically independent, we have the core equation: Ψ(u) =

P

ℓ=1

ϕℓ(uTH) Take 3rd derivatives at point u: Tijk(u) =

P

ℓ=1

Hiℓ Hjℓ Hkℓ Cℓℓℓ(u) (1) At u = 0 ➽ symmetric decomposition of Tijk [HOS] At u = 0 ➽ [Taleb, Comon-Rajih, Yeredor]

Pierre Comon LVA/ICA – Sept. 2010 3 / 43

SLIDE 12

Intro General Exact CP Approx CP NonNeg Coherence AAP END

Tensors: General items

Pierre Comon LVA/ICA – Sept. 2010 4 / 43

SLIDE 13

Intro General Exact CP Approx CP NonNeg Coherence AAP END Notation

Arrays and Multi-linearity

A tensor of order d is a multi-linear map: S∗

1 ⊗

⊗ ⊗ . . . ⊗ ⊗ ⊗ S∗

m → Sm+1 ⊗

⊗ ⊗ . . . ⊗ ⊗ ⊗ Sd

Once bases of spaces Sℓ are fixed, they can be represented by d-way arrays of coordinates ➽ bilinear form, or linear operator: represented by a matrix ➽ trilinear form, or bilinear operator: by a 3rd order tensor.

Pierre Comon LVA/ICA – Sept. 2010 5 / 43

SLIDE 14

Intro General Exact CP Approx CP NonNeg Coherence AAP END Notation

Arrays and Multi-linearity

A tensor of order d is a multi-linear map: S∗

1 ⊗

⊗ ⊗ . . . ⊗ ⊗ ⊗ S∗

m → Sm+1 ⊗

⊗ ⊗ . . . ⊗ ⊗ ⊗ Sd

Once bases of spaces Sℓ are fixed, they can be represented by d-way arrays of coordinates ➽ bilinear form, or linear operator: represented by a matrix ➽ trilinear form, or bilinear operator: by a 3rd order tensor.

Pierre Comon LVA/ICA – Sept. 2010 5 / 43

SLIDE 15

Intro General Exact CP Approx CP NonNeg Coherence AAP END Notation

Multi-linearity

Compact notation Linear change in contravariant spaces: T ′

ijk

=

npq

AinBjpCkqTnpq Denoted compactly T ′ = (A, B, C) · T (2) Example: covariance matrix z = Ax ⇒ Rz = (A, A) · Rx = A RxAT

Pierre Comon LVA/ICA – Sept. 2010 6 / 43

SLIDE 16

Intro General Exact CP Approx CP NonNeg Coherence AAP END Definitions

Decomposable tensor

A dth order “decomposable” tensor is the tensor product of d vectors: T = u ⊗

⊗ ⊗ v ⊗ ⊗ ⊗ . . . ⊗ ⊗ ⊗ w

and has coordinates Tij...k = ui vj . . . wk. may be seen as a discretization of multivariate functions whose variables separate: t(x, y, . . . , z) = u(x) v(y) . . . w(z) Nothing else but rank-1 tensors, with forthcoming definition

Pierre Comon LVA/ICA – Sept. 2010 7 / 43

SLIDE 17

Intro General Exact CP Approx CP NonNeg Coherence AAP END Definitions

Decomposable tensor

A dth order “decomposable” tensor is the tensor product of d vectors: T = u ⊗

⊗ ⊗ v ⊗ ⊗ ⊗ . . . ⊗ ⊗ ⊗ w

and has coordinates Tij...k = ui vj . . . wk. may be seen as a discretization of multivariate functions whose variables separate: t(x, y, . . . , z) = u(x) v(y) . . . w(z) Nothing else but rank-1 tensors, with forthcoming definition

Pierre Comon LVA/ICA – Sept. 2010 7 / 43

SLIDE 18

Intro General Exact CP Approx CP NonNeg Coherence AAP END Definitions

Decomposable tensor

A dth order “decomposable” tensor is the tensor product of d vectors: T = u ⊗

⊗ ⊗ v ⊗ ⊗ ⊗ . . . ⊗ ⊗ ⊗ w

and has coordinates Tij...k = ui vj . . . wk. may be seen as a discretization of multivariate functions whose variables separate: t(x, y, . . . , z) = u(x) v(y) . . . w(z) Nothing else but rank-1 tensors, with forthcoming definition

Pierre Comon LVA/ICA – Sept. 2010 7 / 43

SLIDE 19

Intro General Exact CP Approx CP NonNeg Coherence AAP END Definitions

Example

Take v =

1

−1

Then

v⊗

⊗ ⊗ 3 def

= v ⊗

⊗ ⊗ v ⊗ ⊗ ⊗ v =

1

−1 −1 1 −1 1 1 −1

This is a “decomposable” symmetric tensor → rank-1

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

✇ ✇ ✇ ✇ ✇ ✇ ✇ ✇

blue bullets = 1, red bullets = −1.

Pierre Comon LVA/ICA – Sept. 2010 8 / 43

SLIDE 20

Intro General Exact CP Approx CP NonNeg Coherence AAP END Definitions

Example

Take v =

1

−1

Then

v⊗

⊗ ⊗ 3 def

= v ⊗

⊗ ⊗ v ⊗ ⊗ ⊗ v =

1

−1 −1 1 −1 1 1 −1

This is a “decomposable” symmetric tensor → rank-1

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

✇ ✇ ✇ ✇ ✇ ✇ ✇ ✇

blue bullets = 1, red bullets = −1.

Pierre Comon LVA/ICA – Sept. 2010 8 / 43

SLIDE 21

Intro General Exact CP Approx CP NonNeg Coherence AAP END Definitions

Example

Take v =

1

−1

Then

v⊗

⊗ ⊗ 3 def

= v ⊗

⊗ ⊗ v ⊗ ⊗ ⊗ v =

1

−1 −1 1 −1 1 1 −1

This is a “decomposable” symmetric tensor → rank-1

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

✇ ✇ ✇ ✇ ✇ ✇ ✇ ✇

blue bullets = 1, red bullets = −1.

Pierre Comon LVA/ICA – Sept. 2010 8 / 43

SLIDE 22

Intro General Exact CP Approx CP NonNeg Coherence AAP END Definitions

From SVD to tensor decompositions

Matrix SVD, M = (U, V) · Σ, may be extended in at least two ways to tensors Keep orthogonality: Orthogonal Tucker, HOSVD T = (U, V, W) · C C is R1 × R2 × R3: multilinear rank = (R1, R2, R3) Keep diagonality: Canonical Polyadic decomposition (CP) T = (A, B, C) · L L is R × R × R diagonal, λi = 0: rank = R.

Pierre Comon LVA/ICA – Sept. 2010 9 / 43

SLIDE 23

Intro General Exact CP Approx CP NonNeg Coherence AAP END Definitions

From SVD to tensor decompositions

Matrix SVD, M = (U, V) · Σ, may be extended in at least two ways to tensors Keep orthogonality: Orthogonal Tucker, HOSVD T = (U, V, W) · C C is R1 × R2 × R3: multilinear rank = (R1, R2, R3) Keep diagonality: Canonical Polyadic decomposition (CP) T = (A, B, C) · L L is R × R × R diagonal, λi = 0: rank = R.

Pierre Comon LVA/ICA – Sept. 2010 9 / 43

SLIDE 24

Intro General Exact CP Approx CP NonNeg Coherence AAP END Definitions

From SVD to tensor decompositions

Matrix SVD, M = (U, V) · Σ, may be extended in at least two ways to tensors Keep orthogonality: Orthogonal Tucker, HOSVD T = (U, V, W) · C C is R1 × R2 × R3: multilinear rank = (R1, R2, R3) Keep diagonality: Canonical Polyadic decomposition (CP) T = (A, B, C) · L L is R × R × R diagonal, λi = 0: rank = R.

Pierre Comon LVA/ICA – Sept. 2010 9 / 43

SLIDE 25

Intro General Exact CP Approx CP NonNeg Coherence AAP END Definitions

From SVD to tensor decompositions

Matrix SVD, M = (U, V) · Σ, may be extended in at least two ways to tensors Keep orthogonality: Orthogonal Tucker, HOSVD T = (U, V, W) · C C is R1 × R2 × R3: multilinear rank = (R1, R2, R3) Keep diagonality: Canonical Polyadic decomposition (CP) T = (A, B, C) · L L is R × R × R diagonal, λi = 0: rank = R.

Pierre Comon LVA/ICA – Sept. 2010 9 / 43

SLIDE 26

Intro General Exact CP Approx CP NonNeg Coherence AAP END Definitions

From SVD to tensor decompositions

Matrix SVD, M = (U, V) · Σ, may be extended in at least two ways to tensors Keep orthogonality: Orthogonal Tucker, HOSVD T = (U, V, W) · C C is R1 × R2 × R3: multilinear rank = (R1, R2, R3) Keep diagonality: Canonical Polyadic decomposition (CP) T = (A, B, C) · L L is R × R × R diagonal, λi = 0: rank = R.

Pierre Comon LVA/ICA – Sept. 2010 9 / 43

SLIDE 27

Intro General Exact CP Approx CP NonNeg Coherence AAP END

Exact Canonical Polyadic (CP) decomposition

Pierre Comon LVA/ICA – Sept. 2010 10 / 43

SLIDE 28

Intro General Exact CP Approx CP NonNeg Coherence AAP END CP definition

Canonical Polyadic (CP) decomposition

Any I × J × · · · × K tensor T can be decomposed as T =

q

λq u(q) ⊗

⊗ ⊗ v(q) ⊗ ⊗ ⊗ . . . ⊗ ⊗ ⊗ w(q)

➽ “Polyadic form” [Hitchcock’27] The tensor rank of T is the minimal number R(T ) of “decomposable” terms such that equality holds. May impose unit norm vectors u(q), v(q), . . . w(q)

Pierre Comon LVA/ICA – Sept. 2010 11 / 43

SLIDE 29

Intro General Exact CP Approx CP NonNeg Coherence AAP END CP definition

Canonical Polyadic (CP) decomposition

Any I × J × · · · × K tensor T can be decomposed as T =

R(T)

q

λq u(q) ⊗

⊗ ⊗ v(q) ⊗ ⊗ ⊗ . . . ⊗ ⊗ ⊗ w(q)

➽ “Polyadic form” [Hitchcock’27] The tensor rank of T is the minimal number R(T ) of “decomposable” terms such that equality holds. May impose unit norm vectors u(q), v(q), . . . w(q)

Pierre Comon LVA/ICA – Sept. 2010 11 / 43

SLIDE 30

Intro General Exact CP Approx CP NonNeg Coherence AAP END CP definition

Canonical Polyadic (CP) decomposition

Any I × J × · · · × K tensor T can be decomposed as T =

R(T)

q

λq u(q) ⊗

⊗ ⊗ v(q) ⊗ ⊗ ⊗ . . . ⊗ ⊗ ⊗ w(q)

➽ “Polyadic form” [Hitchcock’27] The tensor rank of T is the minimal number R(T ) of “decomposable” terms such that equality holds. May impose unit norm vectors u(q), v(q), . . . w(q)

Pierre Comon LVA/ICA – Sept. 2010 11 / 43

SLIDE 31

Intro General Exact CP Approx CP NonNeg Coherence AAP END CP definition

Hitchcock

Frank Lauren Hitchcock (1875-1957)

[Courtesy of L-H.Lim]

Pierre Comon LVA/ICA – Sept. 2010 12 / 43

SLIDE 32

Intro General Exact CP Approx CP NonNeg Coherence AAP END CP definition

Hitchcock

Frank Lauren Hitchcock (1875-1957)

[Courtesy of L-H.Lim]

Claude Elwood Shannon (1916-2001)

Pierre Comon LVA/ICA – Sept. 2010 12 / 43

SLIDE 33

Intro General Exact CP Approx CP NonNeg Coherence AAP END CP definition

Towards a unique terminology

Minimal Polyadic Form [Hitchcock’27] Canonical decomposition [Weinstein’84, Carroll’70, Chiantini-Ciliberto’06, Comon’00, Khoromskij, Tyrtyshnikov] Parafac [Harshman’70, Sidiropoulos’00] Optimal computation [Strassen’83] Minimum-length additive decomposition (AD) [Iarrobino’96] Suggestion: Canonical Polyadic decomposition (CP) [Comon’08, Grasedyk, Espig...] CP does also already stand for Candecomp/Parafac [Bro’97, Kiers’98, tenBerge’04...]

Pierre Comon LVA/ICA – Sept. 2010 13 / 43

SLIDE 34

Intro General Exact CP Approx CP NonNeg Coherence AAP END CP definition

Psychometrics

Richard A. Harshman

(1970)

J. Douglas Carroll

(1970)

Pierre Comon LVA/ICA – Sept. 2010 14 / 43

SLIDE 35

Intro General Exact CP Approx CP NonNeg Coherence AAP END CP definition

Uniqueness: Kruskal 1/2

The Kruskal rank of a matrix A is the maximum number kA, such that any subset of kA columns are linearly independent.

Pierre Comon LVA/ICA – Sept. 2010 15 / 43

SLIDE 36

Intro General Exact CP Approx CP NonNeg Coherence AAP END CP definition

Uniqueness: Kruskal 2/2

Sufficient condition for uniqueness of CP [Kruskal’77, Sidiropoulos-Bro’00, Landsberg’09]: Essential uniqueness is ensured if tensor rank R is below the so-called Kruskal’s bound: 2R + 2 ≤ kA + kB + kC (3)

r generically, for a tensor of order d and dimensions Nℓ:

2R ≤

d

ℓ=1

min(Nℓ, R) − d + 1 ➽ Bound much smaller than expected rank: ∃ a much better bound, in almost sure sense

Pierre Comon LVA/ICA – Sept. 2010 16 / 43

SLIDE 37

Intro General Exact CP Approx CP NonNeg Coherence AAP END CP definition

Uniqueness: Kruskal 2/2

Sufficient condition for uniqueness of CP [Kruskal’77, Sidiropoulos-Bro’00, Landsberg’09]: Essential uniqueness is ensured if tensor rank R is below the so-called Kruskal’s bound: 2R + 2 ≤ kA + kB + kC (3)

r generically, for a tensor of order d and dimensions Nℓ:

2R ≤

d

ℓ=1

min(Nℓ, R) − d + 1 ➽ Bound much smaller than expected rank: ∃ a much better bound, in almost sure sense

Pierre Comon LVA/ICA – Sept. 2010 16 / 43

SLIDE 38

Intro General Exact CP Approx CP NonNeg Coherence AAP END CP definition

Rank-3 example 1/2

Example

T =

+ 2 2 = +

Pierre Comon LVA/ICA – Sept. 2010 17 / 43

SLIDE 39

Intro General Exact CP Approx CP NonNeg Coherence AAP END CP definition

Rank-3 example 1/2

Example

T =

+ 2 2 = + = + + 2

blue bullets = 1, red bullets = −1.

Pierre Comon LVA/ICA – Sept. 2010 17 / 43

SLIDE 40

Intro General Exact CP Approx CP NonNeg Coherence AAP END CP definition

Rank-3 example 2/2

Conclusion: the 2 × 2 × 2 tensor T = 2 1 1 1

admits the CP

T = 1 1 ⊗

⊗ ⊗ 3

+ −1 1 ⊗

⊗ ⊗ 3

+ 2

−1

⊗

⊗ ⊗ 3

and has rank 3, hence larger than dimension

Pierre Comon LVA/ICA – Sept. 2010 18 / 43

SLIDE 41

Intro General Exact CP Approx CP NonNeg Coherence AAP END

Approximate Canonical Polyadic (CP) decomposition

Pierre Comon LVA/ICA – Sept. 2010 19 / 43

SLIDE 42

Intro General Exact CP Approx CP NonNeg Coherence AAP END Motivation

Why need for approximation?

Additive noise in measurements Noise has a continuous probability distribution Then tensor rank is generic Hence there are often infinitely many CP decompositions ➽ Approximations aim at getting rid of noise, and at restoring uniqueness: Arg inf

a(p),b(p),c(p) ||T − r

p=1

a(p) ⊗

⊗ ⊗ b(p) . . . ⊗ ⊗ ⊗ c(p)||2

(4) But infimum may be reached for tensors of rank > r !

Pierre Comon LVA/ICA – Sept. 2010 20 / 43

SLIDE 43

Intro General Exact CP Approx CP NonNeg Coherence AAP END Motivation

Why need for approximation?

Additive noise in measurements Noise has a continuous probability distribution Then tensor rank is generic Hence there are often infinitely many CP decompositions ➽ Approximations aim at getting rid of noise, and at restoring uniqueness: Arg inf

a(p),b(p),c(p) ||T − r

p=1

a(p) ⊗

⊗ ⊗ b(p) . . . ⊗ ⊗ ⊗ c(p)||2

(4) But infimum may be reached for tensors of rank > r !

Pierre Comon LVA/ICA – Sept. 2010 20 / 43

SLIDE 44

Intro General Exact CP Approx CP NonNeg Coherence AAP END Border rank

Border rank

T has border rank R iff it is the limit of tensors of rank R, and not the limit of tensors of lower rank. [Bini’79, Sch¨

nhage’81, Strassen’83, Likteig’85]

rank > R

+

x

+ + + + + +

+

rank ≤ R

Pierre Comon LVA/ICA – Sept. 2010 21 / 43

SLIDE 45

Intro General Exact CP Approx CP NonNeg Coherence AAP END Border rank

Example

Let u and v be non collinear vectors. Define T0 [Comon et al.’08]: T0 = u ⊗

⊗ ⊗ u ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ v + u ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ v ⊗ ⊗ ⊗ u + u ⊗ ⊗ ⊗ v ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ u + v ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ u

And define sequence Tε = 1

ε

(u + ε v)⊗

⊗ ⊗ 4 − u⊗ ⊗ ⊗ 4

. Then Tε → T0 as ε → 0 ➽ Hence rank{T0} = 4, but rank{T0} = 2

Pierre Comon LVA/ICA – Sept. 2010 22 / 43

SLIDE 46

Intro General Exact CP Approx CP NonNeg Coherence AAP END Border rank

Example

Let u and v be non collinear vectors. Define T0 [Comon et al.’08]: T0 = u ⊗

⊗ ⊗ u ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ v + u ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ v ⊗ ⊗ ⊗ u + u ⊗ ⊗ ⊗ v ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ u + v ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ u

And define sequence Tε = 1

ε

(u + ε v)⊗

⊗ ⊗ 4 − u⊗ ⊗ ⊗ 4

. Then Tε → T0 as ε → 0 ➽ Hence rank{T0} = 4, but rank{T0} = 2

Pierre Comon LVA/ICA – Sept. 2010 22 / 43

SLIDE 47

Intro General Exact CP Approx CP NonNeg Coherence AAP END Border rank

Example

Let u and v be non collinear vectors. Define T0 [Comon et al.’08]: T0 = u ⊗

⊗ ⊗ u ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ v + u ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ v ⊗ ⊗ ⊗ u + u ⊗ ⊗ ⊗ v ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ u + v ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ u

And define sequence Tε = 1

ε

(u + ε v)⊗

⊗ ⊗ 4 − u⊗ ⊗ ⊗ 4

. Then Tε → T0 as ε → 0 ➽ Hence rank{T0} = 4, but rank{T0} = 2

Pierre Comon LVA/ICA – Sept. 2010 22 / 43

SLIDE 48

Intro General Exact CP Approx CP NonNeg Coherence AAP END Border rank

Example

Let u and v be non collinear vectors. Define T0 [Comon et al.’08]: T0 = u ⊗

⊗ ⊗ u ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ v + u ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ v ⊗ ⊗ ⊗ u + u ⊗ ⊗ ⊗ v ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ u + v ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ u ⊗ ⊗ ⊗ u

And define sequence Tε = 1

ε

(u + ε v)⊗

⊗ ⊗ 4 − u⊗ ⊗ ⊗ 4

. Then Tε → T0 as ε → 0 ➽ Hence rank{T0} = 4, but rank{T0} = 2

Pierre Comon LVA/ICA – Sept. 2010 22 / 43

SLIDE 49

Intro General Exact CP Approx CP NonNeg Coherence AAP END Border rank

Ill-posedness

Tensors for which rank{T } < rank{T } are such that the approximating sequence contains several decomposable tensors which tend to infinity and cancel each other, viz, some columns become close to collinear Ideas towards a well-posed problem: Prevent collinearity or bound columns.

Pierre Comon LVA/ICA – Sept. 2010 23 / 43

SLIDE 50

Intro General Exact CP Approx CP NonNeg Coherence AAP END Border rank

Ill-posedness

Tensors for which rank{T } < rank{T } are such that the approximating sequence contains several decomposable tensors which tend to infinity and cancel each other, viz, some columns become close to collinear Ideas towards a well-posed problem: Prevent collinearity or bound columns.

Pierre Comon LVA/ICA – Sept. 2010 23 / 43

SLIDE 51

Intro General Exact CP Approx CP NonNeg Coherence AAP END Border rank

Ill-posedness

Tensors for which rank{T } < rank{T } are such that the approximating sequence contains several decomposable tensors which tend to infinity and cancel each other, viz, some columns become close to collinear Ideas towards a well-posed problem: Prevent collinearity or bound columns.

Pierre Comon LVA/ICA – Sept. 2010 23 / 43

SLIDE 52

Intro General Exact CP Approx CP NonNeg Coherence AAP END Border rank

Ill-posedness

Tensors for which rank{T } < rank{T } are such that the approximating sequence contains several decomposable tensors which tend to infinity and cancel each other, viz, some columns become close to collinear Ideas towards a well-posed problem: Prevent collinearity or bound columns.

Pierre Comon LVA/ICA – Sept. 2010 23 / 43

SLIDE 53

Intro General Exact CP Approx CP NonNeg Coherence AAP END Existence

Remedies

1 Impose orthogonality of columns within factor matrices

[Comon’92]

2 Impose orthogonality between decomposable tensors

[Kolda’01]

3 Prevent tendency to infinity by norm constraint on factor

matrices [Paatero’00]

4 Nonnegative tensors: impose decomposable tensors to be

nonnegative [Lim-Comon’09] → “nonnegative rank”

5 Impose minimal angle between columns of factor matrices

[Lim-Comon’10]

Pierre Comon LVA/ICA – Sept. 2010 24 / 43

SLIDE 54

Intro General Exact CP Approx CP NonNeg Coherence AAP END Existence

Remedies

1 Impose orthogonality of columns within factor matrices

[Comon’92]

2 Impose orthogonality between decomposable tensors

[Kolda’01]

3 Prevent tendency to infinity by norm constraint on factor

matrices [Paatero’00]

4 Nonnegative tensors: impose decomposable tensors to be

nonnegative [Lim-Comon’09] → “nonnegative rank”

5 Impose minimal angle between columns of factor matrices

[Lim-Comon’10]

Pierre Comon LVA/ICA – Sept. 2010 24 / 43

SLIDE 55

Intro General Exact CP Approx CP NonNeg Coherence AAP END

Nonnegativity constraint: example

Pierre Comon LVA/ICA – Sept. 2010 25 / 43

SLIDE 56

Intro General Exact CP Approx CP NonNeg Coherence AAP END Fluo

Fluorescence Spectroscopy 1/3

An optical excitation produces several effects Rayleigh diffusion Raman diffusion Fluorescence At low concentrations, Beer-Lambert law can be linearized [Luciani’09] I(λf , λe, k) = Io

ℓ

γℓ(λf ) ǫℓ(λe) ck,ℓ ➽ Hence 3rd array decomposition with real nonnegative factors [Bro’97].

Pierre Comon LVA/ICA – Sept. 2010 26 / 43

SLIDE 57

Intro General Exact CP Approx CP NonNeg Coherence AAP END Fluo

Fluorescence Spectroscopy 1/3

An optical excitation produces several effects Rayleigh diffusion Raman diffusion Fluorescence At low concentrations, Beer-Lambert law can be linearized [Luciani’09] I(λf , λe, k) = Io

ℓ

γℓ(λf ) ǫℓ(λe) ck,ℓ ➽ Hence 3rd array decomposition with real nonnegative factors [Bro’97].

Pierre Comon LVA/ICA – Sept. 2010 26 / 43

SLIDE 58

Intro General Exact CP Approx CP NonNeg Coherence AAP END Fluo

Fluorescence Spectroscopy 1/3

An optical excitation produces several effects Rayleigh diffusion Raman diffusion Fluorescence At low concentrations, Beer-Lambert law can be linearized [Luciani’09] I(λf , λe, k) = Io

ℓ

γℓ(λf ) ǫℓ(λe) ck,ℓ ➽ Hence 3rd array decomposition with real nonnegative factors [Bro’97].

Pierre Comon LVA/ICA – Sept. 2010 26 / 43

SLIDE 59

Intro General Exact CP Approx CP NonNeg Coherence AAP END Fluo

Fluorescence Spectroscopy 2/3

Mixture of 4 solutes (one concentration shown)

Pierre Comon LVA/ICA – Sept. 2010 27 / 43

SLIDE 60

Intro General Exact CP Approx CP NonNeg Coherence AAP END Fluo

Fluorescence Spectroscopy 3/3

Obtained results with R = 4

Pierre Comon LVA/ICA – Sept. 2010 28 / 43

SLIDE 61

Intro General Exact CP Approx CP NonNeg Coherence AAP END

Approximate CP decomposition: Coherence constraint

Pierre Comon LVA/ICA – Sept. 2010 29 / 43

SLIDE 62

Intro General Exact CP Approx CP NonNeg Coherence AAP END Coherence

Coherence

Definition: [Donoho’03, Gribonval’03, Cand` es’07] let A a matrix with unit-norm columns, ap. µA = max

p=q |ap, aq|

(5) this “coherence of A” is used in Sparse Representation theory

Pierre Comon LVA/ICA – Sept. 2010 30 / 43

SLIDE 63

Intro General Exact CP Approx CP NonNeg Coherence AAP END Existence: coercivity

Existence

Best rank-R approximate under angular constraint Proposition: [Lim-Comon’2010] Let L diagonal, and A, B and C have R unit norm columns. If R < [µAµBµC]−1, then: inf

A,B,C ||T − (A, B, C) · L||

is attained.

Pierre Comon LVA/ICA – Sept. 2010 31 / 43

SLIDE 64

Intro General Exact CP Approx CP NonNeg Coherence AAP END Uniqueness

Uniqueness: Back to Kruskal

Lemma: (spark) [Gribonval’03] kA ≥ 1 µA

Pierre Comon LVA/ICA – Sept. 2010 32 / 43

SLIDE 65

Intro General Exact CP Approx CP NonNeg Coherence AAP END Uniqueness

Uniqueness

Proposition: [Lim-Comon’10] If T = (A, B, C) · L, with λp = 0 for 1 ≤ p ≤ R, A, B, C matrices with unit norm columns, and: 2R + 2 ≤ 1 µA + 1 µB + 1 µC (6) then T has a unique CP decomposition of rank R, up to unit modulus scalar factors (ρA, ρB, ρC), ρAρBρC = 1. Hence it suffices that one µ is small, not every

Pierre Comon LVA/ICA – Sept. 2010 33 / 43

SLIDE 66

Intro General Exact CP Approx CP NonNeg Coherence AAP END Uniqueness

Uniqueness

Proposition: [Lim-Comon’10] If T = (A, B, C) · L, with λp = 0 for 1 ≤ p ≤ R, A, B, C matrices with unit norm columns, and: 2R + 2 ≤ 1 µA + 1 µB + 1 µC (6) then T has a unique CP decomposition of rank R, up to unit modulus scalar factors (ρA, ρB, ρC), ρAρBρC = 1. Hence it suffices that one µ is small, not every

Pierre Comon LVA/ICA – Sept. 2010 33 / 43

SLIDE 67

Intro General Exact CP Approx CP NonNeg Coherence AAP END

Angular constraint: example

Pierre Comon LVA/ICA – Sept. 2010 34 / 43

SLIDE 68

Intro General Exact CP Approx CP NonNeg Coherence AAP END MIMO

Antenna Array Processing

Transmitter Receiver

Pierre Comon LVA/ICA – Sept. 2010 35 / 43

SLIDE 69

Intro General Exact CP Approx CP NonNeg Coherence AAP END MIMO

Antenna Array Processing

Transmitter Receiver

Pierre Comon LVA/ICA – Sept. 2010 35 / 43

SLIDE 70

Intro General Exact CP Approx CP NonNeg Coherence AAP END MIMO

Antenna Array Processing

Receiver Transmitter

Pierre Comon LVA/ICA – Sept. 2010 35 / 43

SLIDE 71

Intro General Exact CP Approx CP NonNeg Coherence AAP END MIMO

Antenna Array Processing

Receiver Transmitter

Pierre Comon LVA/ICA – Sept. 2010 35 / 43

SLIDE 72

Intro General Exact CP Approx CP NonNeg Coherence AAP END MIMO

Antenna Array Processing

Receiver Transmitter

Pierre Comon LVA/ICA – Sept. 2010 35 / 43

SLIDE 73

Intro General Exact CP Approx CP NonNeg Coherence AAP END MIMO

Antenna Array Processing

Receiver Transmitter

Pierre Comon LVA/ICA – Sept. 2010 35 / 43

SLIDE 74

Intro General Exact CP Approx CP NonNeg Coherence AAP END MIMO

Narrow band model in the far field

Modeling the signals received on an array of antennas generally leads to a matrix decomposition: Tij =

q

aiq sjq i: space q: path, source A: antenna gains j: time S: transmitted signals But in the presence of additional diversity, a tensor can be constructed, thanks to new index p

Pierre Comon LVA/ICA – Sept. 2010 36 / 43

SLIDE 75

Intro General Exact CP Approx CP NonNeg Coherence AAP END MIMO

Narrow band model in the far field

Modeling the signals received on an array of antennas generally leads to a matrix decomposition: Tijp =

q

aiq sjq hpq i: space q: path, source A: antenna gains j: time S: transmitted signals But in the presence of additional diversity, a tensor can be constructed, thanks to new index p

Pierre Comon LVA/ICA – Sept. 2010 36 / 43

SLIDE 76

Intro General Exact CP Approx CP NonNeg Coherence AAP END MIMO

Possible diversities in Signal Processing

space time space translation (array geometry) time translation (chip) frequency/wavenumber (nonstationarity) excess bandwidth (oversampling) cyclostationarity polarization finite alphabet ...

Pierre Comon LVA/ICA – Sept. 2010 37 / 43

SLIDE 77

Intro General Exact CP Approx CP NonNeg Coherence AAP END Geometry

Space translation diversity (1/2)

Pierre Comon LVA/ICA – Sept. 2010 38 / 43

SLIDE 78

Intro General Exact CP Approx CP NonNeg Coherence AAP END Geometry

Space translation diversity (1/2)

Pierre Comon LVA/ICA – Sept. 2010 38 / 43

SLIDE 79

Intro General Exact CP Approx CP NonNeg Coherence AAP END Geometry

Space translation diversity (1/2)

Pierre Comon LVA/ICA – Sept. 2010 38 / 43

SLIDE 80

Intro General Exact CP Approx CP NonNeg Coherence AAP END Geometry

Space translation diversity (1/2)

Pierre Comon LVA/ICA – Sept. 2010 38 / 43

SLIDE 81

Intro General Exact CP Approx CP NonNeg Coherence AAP END Geometry

Space translation diversity (1/2)

Pierre Comon LVA/ICA – Sept. 2010 38 / 43

SLIDE 82

Intro General Exact CP Approx CP NonNeg Coherence AAP END Geometry

Space translation diversity (1/2)

Pierre Comon LVA/ICA – Sept. 2010 38 / 43

SLIDE 83

Intro General Exact CP Approx CP NonNeg Coherence AAP END Geometry

Space translation diversity (1/2)

Pierre Comon LVA/ICA – Sept. 2010 38 / 43

SLIDE 84

Intro General Exact CP Approx CP NonNeg Coherence AAP END Geometry

Space translation diversity (1/2)

Pierre Comon LVA/ICA – Sept. 2010 38 / 43

SLIDE 85

Intro General Exact CP Approx CP NonNeg Coherence AAP END Geometry

Space translation diversity (2/2)

Aiq: gain between sensor i and source q Hpq: transfer between reference and subarray p Sjq: sample j of source q βq: path loss, dq: DOA, bi: sensor location Tensor model (NB far field) [Sidiropoulos’00] Reference subarray: Aiq = βq exp( ω

C bT i dq)

Space translation (from reference subarray): βq exp( ω

C [bi + ∆p]T dq) def

= Aiq Hpq Trilinear model: Tijp =

q

Aiq Sjq Hpq p: index of subarray

Pierre Comon LVA/ICA – Sept. 2010 39 / 43

SLIDE 86

Intro General Exact CP Approx CP NonNeg Coherence AAP END Geometry

Space translation diversity (2/2)

Aiq: gain between sensor i and source q Hpq: transfer between reference and subarray p Sjq: sample j of source q βq: path loss, dq: DOA, bi: sensor location Tensor model (NB far field) [Sidiropoulos’00] Reference subarray: Aiq = βq exp( ω

C bT i dq)

Space translation (from reference subarray): βq exp( ω

C [bi + ∆p]T dq) def

= Aiq Hpq Trilinear model: Tijp =

q

Aiq Sjq Hpq p: index of subarray

Pierre Comon LVA/ICA – Sept. 2010 39 / 43

SLIDE 87

Intro General Exact CP Approx CP NonNeg Coherence AAP END Geometry

Space translation diversity (2/2)

Aiq: gain between sensor i and source q Hpq: transfer between reference and subarray p Sjq: sample j of source q βq: path loss, dq: DOA, bi: sensor location Tensor model (NB far field) [Sidiropoulos’00] Reference subarray: Aiq = βq exp( ω

C bT i dq)

Space translation (from reference subarray): βq exp( ω

C [bi + ∆p]T dq) def

= Aiq Hpq Trilinear model: Tijp =

q

Aiq Sjq Hpq p: index of subarray

Pierre Comon LVA/ICA – Sept. 2010 39 / 43

SLIDE 88

Intro General Exact CP Approx CP NonNeg Coherence AAP END Geometry

Meaning of coherence

1 Unconstrained joint source estimation & localization

[Sidiropoulos’2000] (ill-posed if approximate)

2 Coherence-constrained joint source estimation & localization

[Lim-Comon’2010] time diversity: µA → cross correlation space diversity: µB, µC → angular separation

Pierre Comon LVA/ICA – Sept. 2010 40 / 43

SLIDE 89

Intro General Exact CP Approx CP NonNeg Coherence AAP END Yet other unaddressed topics

Unaddressed topics

spread spectrum communications brain inverse problems medical imaging (MRI) NL filtering noise reduction compression (Tensor trains...) probability hyperspectral imaging structured tensors convolutive mixtures nonnegative factors ... Algorithms

Pierre Comon LVA/ICA – Sept. 2010 41 / 43

SLIDE 90

Intro General Exact CP Approx CP NonNeg Coherence AAP END Why use tensors

QUIZ: Why use tensors?

Main reason: essential uniqueness ➽ Identifiability recovery, up to scale-permutation Sometimes: powerful deterministic approaches Secondary reason: more sources with fewer sensors ➽ Matrices A, B, C may have more columns than rows

Pierre Comon LVA/ICA – Sept. 2010 42 / 43

SLIDE 91

Intro General Exact CP Approx CP NonNeg Coherence AAP END Why use tensors

QUIZ: Why use tensors?

Main reason: essential uniqueness ➽ Identifiability recovery, up to scale-permutation Sometimes: powerful deterministic approaches Secondary reason: more sources with fewer sensors ➽ Matrices A, B, C may have more columns than rows

Pierre Comon LVA/ICA – Sept. 2010 42 / 43

SLIDE 92

Intro General Exact CP Approx CP NonNeg Coherence AAP END Why use tensors

QUIZ: Why use tensors?

Main reason: essential uniqueness ➽ Identifiability recovery, up to scale-permutation Sometimes: powerful deterministic approaches Secondary reason: more sources with fewer sensors ➽ Matrices A, B, C may have more columns than rows

Pierre Comon LVA/ICA – Sept. 2010 42 / 43

SLIDE 93

Intro General Exact CP Approx CP NonNeg Coherence AAP END Why use tensors

QUIZ: Why use tensors?

Main reason: essential uniqueness ➽ Identifiability recovery, up to scale-permutation Sometimes: powerful deterministic approaches Secondary reason: more sources with fewer sensors ➽ Matrices A, B, C may have more columns than rows

Pierre Comon LVA/ICA – Sept. 2010 42 / 43

SLIDE 94

Intro General Exact CP Approx CP NonNeg Coherence AAP END Why use tensors

Other perspectives

Ignorance is the necessary condition for human being happiness. Anatole France (1844-1924)

Pierre Comon LVA/ICA – Sept. 2010 43 / 43

SLIDE 95

Intro General Exact CP Approx CP NonNeg Coherence AAP END Why use tensors

Other perspectives

Ignorance is the necessary condition for human being happiness. Anatole France (1844-1924) Only when the last tree has died, the last river has been poisoned and the last fish has been caught will we realize that we cannot eat money. Cree proverb

Pierre Comon LVA/ICA – Sept. 2010 43 / 43