Axioms for graph clustering Twan van Laarhoven and Elena Marchiori - - PowerPoint PPT Presentation

▶
axioms for graph clustering
SMART_READER_LITE
LIVE PREVIEW

Axioms for graph clustering Twan van Laarhoven and Elena Marchiori - - PowerPoint PPT Presentation

Jan 21, 2023 275 likes •1.07k views

Axioms for graph clustering Twan van Laarhoven and Elena Marchiori Institute for Computing and Information Sciences Radboud University Nijmegen, The Netherlands 27th September 2013 1 / 49 Outline Introduction Axioms for data clustering

slide-1
SLIDE 1

Axioms for graph clustering

Twan van Laarhoven and Elena Marchiori

Institute for Computing and Information Sciences Radboud University Nijmegen, The Netherlands

27th September 2013

1 / 49

slide-2
SLIDE 2

Outline

Introduction Axioms for data clustering Axioms for graph clustering Modularity Conclusion

2 / 49

slide-3
SLIDE 3

Outline

Introduction Axioms for data clustering Axioms for graph clustering Modularity Conclusion

3 / 49

slide-4
SLIDE 4

Clustering

  • Image processing, medicine,
  • biology, economy, ... see, e.g., UCI ML repository.

4 / 49

slide-5
SLIDE 5

Clustering

  • social sciences,
  • life sciences, brain research, ... see, e.g., UCI Network

Data repository.

5 / 49

slide-6
SLIDE 6

Clustering: what is it?

  • Informally: grouping objects in such a way that objects in

each group are more similar to each other than to objects in other groups.

  • Formally: an optimization problem. Define an objective

function whose optimization yields a division of objects into (disjoint) groups. k-means clustering objective:

  • c∈C
  • x∈c

|| x − µc||2, where µc =

  • x∈c

x/|c|.

6 / 49

slide-7
SLIDE 7

Clustering: how to do it?

  • Clustering as an optimization problem is in general

NP-hard.

  • Efficient heuristic and approximation algorithms are

developed to find sub optimal solutions.

7 / 49

slide-8
SLIDE 8

Clustering: data versus graphs

  • Data clustering uses a distance func-

tion that quantifies the similarity be- tween each pair of patterns.

  • Graph clustering uses weighted edges

describing a relation over patterns.

8 / 49

slide-9
SLIDE 9

From data to graph clustering

  • Proximity graphs may be used to transform a data

clustering problem into a graph clustering one.

Distance matrix → kNN graph → Graph clustering      · · · · · · · · · · · · · · · · · · · · · · · ·     

9 / 49

slide-10
SLIDE 10

Outline

Introduction Axioms for data clustering Axioms for graph clustering Modularity Conclusion

10 / 49

slide-11
SLIDE 11

Why axioms?

  • There is no unique definition of clustering.
  • Can we formalize our intuition of good objective

functions?

  • Are existing objective functions good?
  • Can we design better objective functions?

11 / 49

slide-12
SLIDE 12

Axioms for data clustering

Kleinberg’ s axiomatic framework

Kleinberg proved an impossibility result concerning the axiomatization of the notion of data clustering. He focused on clustering functions ˆ C : D → C, from distance functions over a dataset S to clusterings of S, d → C.

Theorem (Kleinberg 2002)

There is no clustering function that is scale invariant, consistent and rich.

12 / 49

slide-13
SLIDE 13

Kleinberg’s axioms

  • Scale-Invariance.

∀d ∈ D, α > 0. ˆ C(d) = ˆ C(αd). ˆ C   a b c d   = ˆ C    a b c d   

13 / 49

slide-14
SLIDE 14

Kleinberg’s axioms

  • Richness.

range( ˆ C) is equal to the set of all partitions of S. ∃d. ˆ C (d) = a b c d e.g. d = a b c d

13 / 49

slide-15
SLIDE 15

Kleinberg’s axioms

  • Consistency.

∀d, d′ ∈ D. ˆ C(d) = C and d′ is a C-transformation of d ⇒ ˆ C(d′) = C

  • .

d′ is a C-transformation of d if ∀i, j ∈ S

  • i ∼C j ⇒ d′(i, j) ≤ d(i, j);
  • i ∼C j ⇒ d′(i, j) ≥ d(i, j).

ˆ C   a b c   = a b c ⇒ ˆ C

  • a

b c

  • = a

b c

13 / 49

slide-16
SLIDE 16

Kleinberg’s axioms

  • Scale-Invariance.

∀d ∈ D, α > 0. ˆ C(d) = ˆ C(αd).

  • Richness.

range( ˆ C) is equal to the set of all partitions of S.

  • Consistency.

∀d, d′ ∈ D. ˆ C(d) = C and d′ is a C-transformation of d ⇒ ˆ C(d′) = C

  • .

d′ is a C-transformation of d if ∀i, j ∈ S

  • i ∼C j ⇒ d′(i, j) ≤ d(i, j);
  • i ∼C j ⇒ d′(i, j) ≥ d(i, j).

13 / 49

slide-17
SLIDE 17

Kleinberg result

C ′ is a refinement of C (C ′ ⊑ C) if ∀c′ ∈ C ′ ∃c ∈ C s.t. c′ ⊆ c. {C1, . . . , Cn} ⊂ C is an antichain if ∀i, j i = j ⇒ Ci ⊑ Cj.

Theorem

If ˆ C is Scale Invariant and Consistent then range( ˆ C) is an antichain. Proof (sketch) Suppose ˆ C is Consistent and Scale Invariant. Let C0 ⊑ C1 in range( ˆ C). Construct d such that ˆ C(d) = C1. Choose α such that d′ = αd and ˆ C(d′) = C0.

14 / 49

slide-18
SLIDE 18

Other results

Quality functions

Ackerman and Ben-David used quality functions Q instead of clustering functions. Q : D × C → R≥0, mapping a distance function and a clustering into a non-negative real number, (d, C) → r.

Theorem (Ackerman, Ben-David 2008)

There is a clustering quality function that is permutation invariant, scale invariant, monotonic and rich. C-index = (s − smin)/(smax − smin), where s =

i∼C j d(i, j),

smin is the sum of the n minimal (over all pairs of patterns) distances, smax is the sum of the n maximal distances, n = |{(i, j) | i ∼C j}|.

15 / 49

slide-19
SLIDE 19

To summarize

  • Previous work on axioms for clustering objective functions

are framed in terms of distance functions.

  • Kleinberg’s impossibility result is for clustering functions.
  • Quality functions are more flexible and allow for

axiomatization of data clustering.

  • What about graph clustering? This is a different -

although related - story ...

16 / 49

slide-20
SLIDE 20

Outline

Introduction Axioms for data clustering Axioms for graph clustering Modularity Conclusion

17 / 49

slide-21
SLIDE 21

Graphs

Distance functions Graphs d(i, j) E(i, j) c a d b c a d b

18 / 49

slide-22
SLIDE 22

Graphs

Distance functions Graphs d(i, j) E(i, j)

  • c

a d b c a d b

18 / 49

slide-23
SLIDE 23

Graphs

a b c e d g f h i j k

A symmetric weighted graph (or network) is a pair (V , E) of

  • a finite set V of nodes, and
  • a function E : V × V → R≥0 of edge weights,

such that E(i, j) = E(j, i) for all i, j ∈ V .

19 / 49

slide-24
SLIDE 24

Graph clustering

a b c e d g f h i j k

A clustering C of a graph G = (V , E) is a partition of its nodes.

19 / 49

slide-25
SLIDE 25

Clustering: formalizations

  • 1. Clustering function

ˆ C : Graph → Clustering ˆ C    a b c e d    = a b c e d

  • 2. Quality function

Q : Graph × Clustering → R

  • 3. Quality relation

· G · ⊆ Clustering × Clustering

20 / 49

slide-26
SLIDE 26

Clustering: formalizations

  • 1. Clustering function

ˆ C : Graph → Clustering

  • 2. Quality function

Q : Graph × Clustering → R Q    a b c e d    = 0.1234

  • 3. Quality relation

· G · ⊆ Clustering × Clustering

20 / 49

slide-27
SLIDE 27

Clustering: formalizations

  • 1. Clustering function

ˆ C : Graph → Clustering

  • 2. Quality function

Q : Graph × Clustering → R

  • 3. Quality relation

· G · ⊆ Clustering × Clustering a b c e d a b c e d

20 / 49

slide-28
SLIDE 28

Some quality functions

  • Connected components
  • Total weight of within cluster edges

Q(G, C) =

  • c∈C

wc

  • Modularity

Q(G, C) =

  • c∈C
  • wc/vV − (vc/vV )2
  • Many more

Q(G, C) =

  • c∈C

−wc log(vc/vV ) · · ·

21 / 49

slide-29
SLIDE 29

Families of quality functions

  • Connected components with threshold
  • Total weight of within cluster edges with penalty

Q(G, C) =

  • c∈C

wc − α|C|

  • Modularity

Qγ

RB(G, C) =

  • c∈C
  • wc/vV − γ(vc/vV )2
  • Many more

Q(G, C) =

  • c∈C

−wc log(vc/α) · · ·

22 / 49

slide-30
SLIDE 30

Axiom 1: Scale invariance

Intuition: The magnitude of the edge weights shouldn’t matter. ˆ C    a b c e d    = ˆ C    a b c e d   

23 / 49

slide-31
SLIDE 31

Axiom 1: Scale invariance

Intuition: The magnitude of the edge weights shouldn’t matter. Q    a b c e d    = Q    a b c e d   

23 / 49

slide-32
SLIDE 32

Axiom 1: Scale invariance

Intuition: The magnitude of the edge weights shouldn’t matter. Q    a b c e d    = αQ    a b c e d   

23 / 49

slide-33
SLIDE 33

Axiom 1: Scale invariance

Intuition: The magnitude of the edge weights shouldn’t matter. Q       ≥ Q      

  • Q

      ≥ Q      

23 / 49

slide-34
SLIDE 34

Axiom 1: Scale invariance

Intuition: The magnitude of the edge weights shouldn’t matter.

A quality function Q is scale invariant if

  • for all graphs G = (V , E),
  • all constants α > 0,

Q(G, C1) ≥ Q(G, C2) if and only if Q(αG, C1) ≥ Q(αG, C2).

23 / 49

slide-35
SLIDE 35

Axiom 2: Permutation invariance

Intuition: Only the edge weights should matter. Q    a b c e d    = Q    z v y x u   

24 / 49

slide-36
SLIDE 36

Axiom 2: Permutation invariance

Intuition: Only the edge weights should matter.

A quality function Q is permutation invariant if Q(G, C) = Q(f (G), f (C)). for all

  • graphs G = (V , E) and
  • all isomorphisms f : V → V ′,

where f is extended to graphs and clusterings in the obvious way.

24 / 49

slide-37
SLIDE 37

Axiom 3: Richness

Intuition:

  • All clusterings must be possible.

So,

  • no trivial quality functions.
  • no fixed number of clusters.

A quality function Q is rich if

  • for all sets V and
  • all partitions C ∗ of V ,

there is

  • a graph G = (V , E)
  • such that C ∗ is the optimal clustering of G.

25 / 49

slide-38
SLIDE 38

Axiom 4: Monotonicity

Intuition: Adding edges inside a cluster or removing edges between clusters does not make the clustering worse. Q    a b c e d    ≥ Q    a b c e d   

26 / 49

slide-39
SLIDE 39

Axiom 4: Monotonicity

Intuition: Adding edges inside a cluster or removing edges between clusters does not make the clustering worse.

Let

  • G = (V , E) and G ′ = (V , E ′) be graphs, and
  • C be a clustering of G and G ′.

Then G ′ is a C-consistent improvement of G if

  • E ′(i, j) ≥ E(i, j) for all i ∼C j and
  • E ′(i, j) ≤ E(i, j) for all i ∼C j.

26 / 49

slide-40
SLIDE 40

Axiom 4: Monotonicity

Intuition: Adding edges inside a cluster or removing edges between clusters does not make the clustering worse.

A quality function Q is monotonic if Q(G ′, C) ≥ Q(G, C). for all

  • graphs G,
  • all clusterings C of G and
  • all C-consistent improvements G ′ of G.

26 / 49

slide-41
SLIDE 41

Axiom 5: Locality

Intuition: Local changes should have local effects. Q    a b c e d    = Q    a b c    + Q    e d   

27 / 49

slide-42
SLIDE 42

Axiom 5: Locality

Intuition: Local changes should have local effects. Q    a b c e d    ≥ Q    a b c e d   

  • Q

   a b c e d    ≥ Q    a b c e d   

27 / 49

slide-43
SLIDE 43

Axiom 5: Locality

Intuition: Local changes should have local effects. Q   a b c · · ·   ≥ Q   a b c · · ·  

  • Q

     a b c · · ·      ≥ Q      a b c · · ·     

27 / 49

slide-44
SLIDE 44

Axiom 5: Locality

Intuition: Local changes should have local effects.

Two graphs G1 and G2 agree on the neighborhood of Va ⊆ V1 ∩ V2 if E1(i, j) = E2(i, j) for all i ∈ Va, j ∈ V1 ∩ V2, and E1(i, j) = 0 for all i ∈ Va, j ∈ V1 \ V2, and E2(i, j) = 0 for all i ∈ Va, j ∈ V2 \ V1. So, for nodes/clusters in Va, all incident edges are the same.

27 / 49

slide-45
SLIDE 45

Axiom 5: Locality

Intuition: Local changes should have local effects.

A quality function Q is local if

  • for all graphs G1 = (V1, E1) and G2 = (V2, E2)

that agree on a set Va and its neighborhood,

  • for all clusterings C1 of V1 \ Va,

C2 of V2 \ Va and Ca, Da of Va. if Q(G1, Ca ∪ C1) ≥ Q(G1, Da ∪ C1) then Q(G2, Ca ∪ C2) ≥ Q(G2, Da ∪ C2).

27 / 49

slide-46
SLIDE 46

Discontinuity is magic

Theorem

There is a graph clustering function that is scale invariant, permutation invariant, monotonic, rich and local.

ˆ Ccoco(G) = the connected components of G Qcoco(G, C) = 1[C are the connected components of G]

  • Doesn’t this contradict Kleinberg’s theorem?
  • No: edge weight = 0 ⇔ distance = ∞.
  • Connected components are unstable.

28 / 49

slide-47
SLIDE 47

Discontinuity is magic

Theorem

There is a graph clustering function that is scale invariant, permutation invariant, monotonic, rich and local.

ˆ Ccoco(G) = the connected components of G Qcoco(G, C) = 1[C are the connected components of G]

  • Doesn’t this contradict Kleinberg’s theorem?
  • No: edge weight = 0 ⇔ distance = ∞.
  • Connected components are unstable.

28 / 49

slide-48
SLIDE 48

Axiom 6: continuity

Intuition:

  • Don’t allow such unstable quality functions.
  • A small change in edge weights should lead to only a

small change in quality.

A quality function Q is continuous if

  • for every ǫ > 0 and
  • every graph G = (V , E)

there exists a δ > 0 such that

  • for every graph G ′ = (V , E ′) and
  • every clustering C of G,

we have E ′ − Emax < δ ⇒ |Q(G ′, C) − Q(G, C)| < ǫ.

29 / 49

slide-49
SLIDE 49

Outline

Introduction Axioms for data clustering Axioms for graph clustering Modularity Conclusion

30 / 49

slide-50
SLIDE 50

Modularity

Intuition:

  • Balance within cluster edges against cluster volume.

Qmodularity(G, C) =

  • i,j∈V

E(i, j) vV − vi vV vj vV

  • 1[i ∼C j].

=

  • c∈C

wc vV − vc vV 2 . Where vc =

  • i∈c
  • j∈V

E(i, j) volume of cluster wc =

  • i∈c
  • j∈c

E(i, j) within cluster weight.

31 / 49

slide-51
SLIDE 51

Properties

The obvious:

  • Modularity is permutation invariant.
  • Modularity is scale invariant.
  • Modularity is continuous.

The less obvious:

  • Modularity is rich.

The bad:

  • Modularity is not local.
  • Modularity is not monotonic.

32 / 49

slide-52
SLIDE 52

Modularity is not local

Qmodularity

  • a

b c d 2 1 2

  • = 0.3

Qmodularity

  • a

b c d 2 1 2

  • = 0

Qmodularity

  • a

b c d x y 2 1 2 20

  • = 0.3

Qmodularity

  • a

b c d x y 2 1 2 20

  • = 0.32

33 / 49

slide-53
SLIDE 53

Modularity is not monotonic

Qmodularity

  • a

b c d 1 1

  • = 0.125

Qmodularity

  • a

b c d 0.1 1

  • = 0.079

Qmodularity

  • a

b c d 1 10

  • = 0.079

34 / 49

slide-54
SLIDE 54

Idea 1: Fix the scale

QM-fixed(G, C) =

  • c∈C

wc M − vc M 2

Is it monotonic? Take vc = wc + bc (within + between) ∂QM-fixed(G, C) ∂wc = 1 M − 2wc + 2bc M2 . This is negative when 2vc > M, so not monotonic.

35 / 49

slide-55
SLIDE 55

Idea 1: Fix the scale

QM-fixed(G, C) =

  • c∈C

wc M − wc + bc M 2

Is it monotonic? Take vc = wc + bc (within + between) ∂QM-fixed(G, C) ∂wc = 1 M − 2wc + 2bc M2 . This is negative when 2vc > M, so not monotonic.

35 / 49

slide-56
SLIDE 56

Idea 2: Add some vc to the denominator

QM,γ(G, C) =

  • c∈C
  • wc

M + γvc −

  • vc

M + γvc 2 .

Adaptive scale modularity is

  • permutation invariant, continuous and local.
  • monotonic for all M ≥ 0 and γ ≥ 2.
  • rich for all M ≥ 0 and γ ≥ 1.
  • scale invariant for M = 0.

36 / 49

slide-57
SLIDE 57

Idea 2: Add some vc to the denominator

QM,γ(G, C) =

  • c∈C
  • wc

M + γvc −

  • vc

M + γvc 2 .

Adaptive scale modularity is

  • permutation invariant, continuous and local.
  • monotonic for all M ≥ 0 and γ ≥ 2.
  • rich for all M ≥ 0 and γ ≥ 1.
  • scale invariant for M = 0.

36 / 49

slide-58
SLIDE 58

Proof of monotonicity

Take partial derivatives (vc = wc + bc) QM,γ(G, C) =

  • c∈C
  • wc

M + γ(wc + bc) −

  • wc + bc

M + γ(wc + bc) 2 . ∂QM,γ(G, C) ∂wc = M2 + (γ − 2)Mwc + (2γ − 2)Mbc + γ2vcbc (M + γvc)3 . ∂QM,γ(G, C) ∂bc = − 2Mvc (M + γvc)3 − γwc (M + γvc)2 ≤ 0. When γ ≥ 2, Q is a monotonic increasing function of wc and decreasing function of bc for all c, so the quality function is monotonic.

37 / 49

slide-59
SLIDE 59

Proof sketch of richness

  • Given a clustering C ∗ take G to be the clique graph of C ∗.
  • Pick edge weight large enough (k > 2|V |3M), then the

effect of M becomes insignificant. Q(G, D) ≈

  • c∈C
  • wd − v 2

d

γvd

  • .
  • There are at most |C ∗| terms in the sum that are > ǫ

(where ǫ depends on k and M)

  • The term for c ∈ C is maximal if c = D, D ⊆ C ∗.

The clique graph with edge weight k of a partition C of V is (V , E) where E(i, j) = k · 1[i ∼C j].

38 / 49

slide-60
SLIDE 60

Related quality functions

  • When γ = 0, we get fixed scale modularity.

Equivalent to other modularity variants.

  • When γ = 0 and M = vV , we get modularity.
  • When M = 0 we get

Q0,γ(G, C) ∝

  • c∈C

wc vc − 1 γ

  • ,

i.e. normalized cut.

  • When M → ∞ we get

Q∞,γ(G, C) ∝

  • c∈C

wc, i.e. unnormalized cut.

39 / 49

slide-61
SLIDE 61

Outline

Introduction Axioms for data clustering Axioms for graph clustering Modularity Conclusion

40 / 49

slide-62
SLIDE 62

Summary

  • Graph and data clustering are related, yet different,

notions.

  • 6 axioms for graph clustering quality functions.
  • Graph setting allows for locality.
  • Modularity is not monotonic.
  • Adaptive scale modularity satisfies all 6 axioms.
  • Generalizes both modularity and normalized cut.
  • Two parameters to control size of clusters.

41 / 49

slide-63
SLIDE 63

Open problems

  • Applications of adaptive scale modularity to real life

problems.

  • Overlapping clusters.
  • Directed graphs.
  • How to use axioms for developing better algorithms for

clustering.

42 / 49

slide-64
SLIDE 64

Thank you for your attention. Axioms for graph clustering

Twan van Laarhoven and Elena Marchiori

Institute for Computing and Information Sciences Radboud University Nijmegen, The Netherlands

27th September 2013

43 / 49

slide-65
SLIDE 65

Extra slides

44 / 49

slide-66
SLIDE 66

Adaptive Scale Modularity behavior

Take a simple graph: w w b

  • Two cliques each with w within weight
  • Connected by edges with total weight b.
  • Total volume 2w + 2b.
  • What is the behavior of adaptive scale modularity?

45 / 49

slide-67
SLIDE 67

M0

3 10 20 30 40 50 10 20 30 40 50 b 1 10 20 30 40 50 10 20 30 40 50 1 2 10 20 30 40 50 10 20 30 40 50 1 2 10 20 30 40 50 10 20 30 40 50

M10

3 10 20 30 40 50 10 20 30 40 50 b 1 2 10 20 30 40 50 10 20 30 40 50 1 2 10 20 30 40 50 10 20 30 40 50 1 2 10 20 30 40 50 10 20 30 40 50

M100

1 2 3 10 20 30 40 50 10 20 30 40 50 b 1 2 10 20 30 40 50 10 20 30 40 50 1 2 10 20 30 40 50 10 20 30 40 50 1 2 10 20 30 40 50 10 20 30 40 50

M1000

1 10 20 30 40 50 10 20 30 40 50 w b 1 10 20 30 40 50 10 20 30 40 50 w 1 10 20 30 40 50 10 20 30 40 50 w 1 2 10 20 30 40 50 10 20 30 40 50 w

Γ 0 Γ 1 Γ 2 Γ 10

Legend:

1 = 2 = 3 =

46 / 49

slide-68
SLIDE 68

Clustering by optimization

  • Graph clustering is NP hard.
  • Top down:

find best cut and repeat

  • Bottom up:

group nodes together

  • Simulated annealing

47 / 49

slide-69
SLIDE 69

Louvain method

  • V.D. Blondel, JL. Guillaume, R. Lambiotte, E. Lefebvre

Fast unfolding of communities in large networks

  • J. Stat. Mech. 2008
  • Best graph clustering method in surveys.
  • Method:

1 Move nodes into neighboring clusters to improve quality. 2 Repeat until local maximum. 3 Now cluster the clusters.

48 / 49

slide-70
SLIDE 70

Louvain method (example)

49 / 49

slide-71
SLIDE 71

Louvain method (example)

49 / 49

slide-72
SLIDE 72

Louvain method (example)

49 / 49

slide-73
SLIDE 73

Louvain method (example)

49 / 49

slide-74
SLIDE 74

Louvain method (example)

49 / 49

slide-75
SLIDE 75

Louvain method (example)

49 / 49

slide-76
SLIDE 76

Louvain method (example)

49 / 49

slide-77
SLIDE 77

Louvain method (example)

49 / 49

slide-78
SLIDE 78

Louvain method (example)

49 / 49

slide-79
SLIDE 79

Louvain method (example)

49 / 49