Clustering Rankings in the Fourier Domain Stphan Clmenon and Romaric - - PowerPoint PPT Presentation

Clustering Rankings in the Fourier Domain

Stéphan Clémençon and Romaric Gaudel and Jérémie Jakubowicz

LTCI, Telecom Paristech (TSI) UMR Institut Telecom/CNRS No. 5141

ECML PKDD, September 2011

Distributions on rankings

Many applications consider ranked data / distributions on rankings

(Uniform distribution with respect to constraints)

◮ Top-k lists ⋆ Rank of the k most preferred objects

3 > 2 > 5 > . . .

◮ Preference data ⋆ Preferences on k (randomly) picked objects

. . . > 3 > . . . > 2 > . . . > 5 > . . . “sushi” dataset

◮ Bucket order ⋆ Preferences on groups of objects

3, 2 > 5, 1, 7 > 4, 6, 8

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 2 / 20

Representation for distributions on Rankings

Probability table

◮ n! (factorial n) coefficients

Fourier representation

[Diaconis, 1989; Kondor & Barbosa, 2010]

◮ n! coefficients ◮ Few relevant coefficients in practice

Parametric models

◮ Mallows

[Mallows, 1957]

◮ Plackett-Luce

[Luce, 1959; Plackett, 1975]

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 3 / 20

Representation for distributions on Rankings

Probability table

◮ n! (factorial n) coefficients

Fourier representation

[Diaconis, 1989; Kondor & Barbosa, 2010]

◮ n! coefficients ◮ Few relevant coefficients in practice

Parametric models

◮ Mallows

[Mallows, 1957]

◮ Plackett-Luce

[Luce, 1959; Plackett, 1975]

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 3 / 20

Contributions

Clustering of rankings through sparse Fourier representation

Position

◮ Clustering of distributions on rankings ⋆ Gather ranking distributions with similar shapes

Proposed approach

◮ Work in the Fourier representation

= ⇒

⋆ Sparse representation of 1 distribution ⋆ Sparse difference between representations of 2 distributions

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 4 / 20

Outline

1

Sparsity in the Fourier Representation

2

Sparse Clustering of Rankings

3

Numerical Experiments

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 5 / 20

Fourier representation

For real line function

Functions are decomposed on the sinusoidal basis

f(x) = 1.1 + 2.1 cos(x) + 3.2 cos(2x) + 1.5 cos(3x) + 0.2 cos(4x) + 0.01 cos(5x) + . . .

= + + + + + The information is contained in few (low frequency) coefficients

= ⇒ Reduced storage/transfer/computation costs

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 6 / 20

Fourier representation

For real line function

Functions are decomposed on the sinusoidal basis

f(x) = 1.1 + 2.1 cos(x) + 3.2 cos(2x) + 1.5 cos(3x) + 0.2 cos(4x) + 0.01 cos(5x) + . . .

= + + + + + The information is contained in few (low frequency) coefficients

= ⇒ Reduced storage/transfer/computation costs

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 6 / 20

Fourier representation

For functions on Sn [Diaconis, 1989]

There is no simple basis (corresponding to eigen-spaces of dimension 1)

= ⇒ Fourier coefficients are matrices indexed by the set Rn of all integer partitions of n Ff =

   ,

, , , , , . . .

  

Rn =

• ξ = (n1, . . . , nk) ∈ N∗k : n1 ≥ · · · ≥ nk,

k

• i=1

ni = n

• , 1 ≤ k ≤ n
• “Low-frequency” coefficients are related to low order summaries

(P[σ(i, j) = (k, ℓ)])

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 7 / 20

Example: Mallows(S5)

Exponential distribution on rankings, γ = 0.1

20 40 60 80 100 120 0.000 0.004 0.008 0.012 [ 3 2 4 1 5 ] [ 3 5 4 2 1 ] [ 1 2 4 5 3 ]

“Temporal” coefficents

20 40 60 80 100 120 −0.06 −0.02 0.02 0.06 [ 3 2 4 1 5 ] [ 3 5 4 2 1 ] [ 1 2 4 5 3 ]

Fourier coefficients Remark:

◮ A few relevant parameters when using the Fourier representation

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 8 / 20

Uncertainty principle

Balancing Sparsity

Theorem

(inspired from [Donoho & Stark, 1989])

Let f ∈ C[Sn] of Fourier transform Ff. Denote by supp(f) = {σ ∈ Sn : f(σ) = 0} and by supp(Ff) = {ξ ∈ Rn : Ff(ξ) = 0} the support of f and that of its Fourier transform respectively. Then, we have: #supp(f) ·

• ξ∈supp(Ff)

d2

ξ ≥ n!.

Direct consequence

◮ Both representations cannot

be simultaneously sparse

0.2 0.4 0.6 0.8 1 20 40 60 80 100 120 distortion # used coefficients γ = 10 γ = 1 γ = 0.1

Distortion with Mallows(S5)

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 9 / 20

Outline

1

Sparsity in the Fourier Representation

2

Sparse Clustering of Rankings

3

Numerical Experiments

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 10 / 20

Clustering of rankings

Aim

◮ Gather distributions on rankings with similar shape

Objective function

◮ Minimize (on all partitions C)

• M(C)

=

L

• l=1
• 1≤i, j≤N

||fi − fj||2 · I{(fi, fj) ∈ C2

l }

= 1 n!

• ξ∈Rn

dξ

L

• l=1
• 1≤i, j≤N: (fi ,fj )∈C2

l

||Ffi(ξ) − Ffj(ξ)||2

HS(dξ)

with dξ × dξ the dimension of the matrix indexed by ξ

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 11 / 20

Managing sparsity

Aim

◮ Gather distributions on rankings with similar shape ◮ Use few Fourier coefficients

New objective function

[Witten & Tibshirani, 2010]

◮ Minimize (on all partitions C, and all weight vectors ω)

• Mω(C) =
• ξ∈Rn

ωξ dξ n!

L

• l=1
• 1≤i, j≤N: (fi ,fj )∈C2

l

||Ffi(ξ) − Ffj(ξ)||2

HS(dξ)

with ω = (ωξ)ξ∈Rn ∈ R#Rn

+

, ||ω||2

l2 ≤ 1 and ||ω||l1 ≤ λ

Remark:

◮ Fixing ω = (1/√#Rn, . . . , 1/√#Rn) leads to the initial optimization

problem (without ω)

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 12 / 20

Algorithm

Initialize ω = (1/√#Rn, . . . , 1/√#Rn) Until convergence, iterate steps 1 and 2

1

Fixing the weight vector ω, minimize Mω(C) after the partition C

2

Fixing the partition C, minimize Mω(C) after ω.

Remarks

◮ Step 1 is performed by a standard clustering algorithm ◮ Step 2 accepts a closed form

[Witten & Tibshirani, 2010]

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 13 / 20

Outline

1

Sparsity in the Fourier Representation

2

Sparse Clustering of Rankings

3

Numerical Experiments

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 14 / 20

Experiments

Aim

◮ Recover clustering information ◮ Use few coefficients

Datasets

◮ Mallows (synthetic) ⋆ Exponential distribution on rankings ◮ Top-k lists (synthetic) ⋆ Uniform distribution on rankings ◮ E-commerce Dataset ⋆ List of purchased products (ordered by date)

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 15 / 20

Mallows(S7)

γ = 1

[ 6 5 1 7 4 2 3 ] [ 6 5 1 7 4 3 2 ] [ 6 5 1 4 7 2 3 ] [ 6 7 3 4 5 1 2 ] [ 4 6 3 7 5 1 2 ] [ 4 7 3 6 5 1 2 ] [ 4 7 3 6 2 1 5 ] [ 3 7 4 6 5 1 2 ] [ 6 5 1 2 4 7 3 ] [ 1 5 6 7 4 2 3 ] 0.00 0.10 0.20

“Temporal” representation (3 coefficients selected)

[ 6 7 3 4 5 1 2 ] [ 3 7 4 6 5 1 2 ] [ 4 6 3 7 5 1 2 ] [ 4 7 3 6 5 1 2 ] [ 4 7 3 6 2 1 5 ] [ 1 5 6 7 4 2 3 ] [ 6 5 1 7 4 3 2 ] [ 6 5 1 2 4 7 3 ] [ 6 5 1 7 4 2 3 ] [ 6 5 1 4 7 2 3 ] 0.02 0.08 0.14

Fourier representation (54 coefficients selected) Remarks:

◮ The Fourier representation recovers the clustering information ◮ The Fourier representation uses few coefficients (compared to n! = 5, 040)

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 16 / 20

Top-4 lists on S8

2 < 1 < 3 < 6 < ... 2 < 1 < 3 < 4 < ... 2 < 1 < 3 < 5 < ... 2 < 1 < 3 < 7 < ... 2 < 1 < 3 < 8 < ... 1 < 3 < 2 < 6 < ... 1 < 3 < 2 < 4 < ... 1 < 3 < 2 < 5 < ... 1 < 3 < 2 < 7 < ... 1 < 3 < 2 < 8 < ... 3 < 2 < 1 < 6 < ... 3 < 2 < 1 < 4 < ... 3 < 2 < 1 < 5 < ... 3 < 2 < 1 < 7 < ... 3 < 2 < 1 < 8 < ... 6 < 8 < 7 < 3 < ... 7 < 8 < 6 < 3 < ... 8 < 6 < 7 < 3 < ... 8 < 7 < 6 < 3 < ... 8 < 7 < 6 < 5 < ... 8 < 7 < 6 < 4 < ... 8 < 6 < 7 < 5 < ... 8 < 6 < 7 < 4 < ... 7 < 8 < 6 < 5 < ... 7 < 8 < 6 < 4 < ... 6 < 8 < 7 < 5 < ... 6 < 8 < 7 < 4 < ... 7 < 6 < 8 < 5 < ... 7 < 6 < 8 < 4 < ... 6 < 7 < 8 < 4 < ... 6 < 7 < 8 < 5 < ... 6 < 7 < 8 < 3 < ... 7 < 6 < 8 < 3 < ... 6 < 8 < 7 < 1 < ... 8 < 6 < 7 < 1 < ... 6 < 8 < 7 < 2 < ... 8 < 6 < 7 < 2 < ... 6 < 7 < 8 < 1 < ... 7 < 6 < 8 < 2 < ... 7 < 8 < 6 < 2 < ... 8 < 7 < 6 < 1 < ... 6 < 7 < 8 < 2 < ... 7 < 6 < 8 < 1 < ... 7 < 8 < 6 < 1 < ... 8 < 7 < 6 < 2 < ... 1 < 2 < 3 < 6 < ... 1 < 2 < 3 < 4 < ... 1 < 2 < 3 < 5 < ... 1 < 2 < 3 < 7 < ... 1 < 2 < 3 < 8 < ... 2 < 3 < 1 < 6 < ... 2 < 3 < 1 < 4 < ... 2 < 3 < 1 < 5 < ... 2 < 3 < 1 < 7 < ... 2 < 3 < 1 < 8 < ... 3 < 1 < 2 < 6 < ... 3 < 1 < 2 < 4 < ... 3 < 1 < 2 < 5 < ... 3 < 1 < 2 < 7 < ... 3 < 1 < 2 < 8 < ... 0.000 0.010 0.020 0.030

Fourier representation (7 coefficients selected) Remarks:

◮ The Fourier representation recovers the clustering information ◮ The “temporal” representation is useless (examples have disjoint supports) ◮ The Fourier representation uses few coefficients

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 17 / 20

E-commerce dataset

0.002 0.006 0.010

Fourier representation (5 coefficients selected) Remarks:

◮ 4 groups among users ◮ Focuses on few coefficients

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 18 / 20

Conclusion and perspectives

Conclusion

A new approach to cluster rankings

◮ Based on the Fourier representation

= ⇒ Sparse representation

◮ Based on a sparse clustering criterion

= ⇒ Focuses on relevant coefficients

Several theoretical and numerical results supporting the approach Future work

◮ Better understanding of the class of distributions with sparse Fourier

representation

• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 19 / 20

Thank you

Bibliography

P . Diaconis A generalization of spectral analysis with application to ranked data. Ann. Stat. 89

• D. Donoho and P

. Stark Uncertainty Principles and Signal Recovery. SIAM J. Appl. Math. 89

• R. Kondor and M. Barbosa Ranking with kernels in Fourier space. COLT’10
• R. D. Luce Individual Choice Behavior. Wiley 59
• C. L. Mallows Non-Null Ranking Models. Biometrika 57
• R. L. Plackett The analysis of permutations. Appl. Stat. 75
• D. Witten A Framework for Feature Selection in Clustering. JASA’10
• S. Clémençon & R. Gaudel & J. Jakubowicz (LTCI)

Clustering Rankings in the Fourier Domain ECML PKDD, September 2011 21 / 20