Supervised Hierarchical Clustering with Exponential Linkage Nishant - - PowerPoint PPT Presentation

Supervised Hierarchical Clustering with Exponential Linkage

Nishant Yadav Ari Kobren Nicholas Monath Andrew McCallum

Supervised Clustering

At train time, learn A : 2X → Y

Supervised Clustering

At train time, learn A : 2X → Y

At test time, use on new set of points

A

Supervised Clustering

At train time, learn A : 2X → Y

At test time, use on new set of points

A

Supervised Clustering

At train time, learn A : 2X → Y

At test time, use on new set of points

A

Select a clustering algorithm Learn a dissimilarity function

Supervised Clustering

At train time, learn A : 2X → Y

At test time, use on new set of points

A

Training Procedure

Clustering Algorithm

Mismatch leads to poor generalization

Select a clustering algorithm Learn a dissimilarity function

Hierarchical Agglomerative Clustering

3

Hierarchical Agglomerative Clustering

3

Hierarchical Agglomerative Clustering

3

Iteratively merge two “closest” clusters

Hierarchical Agglomerative Clustering

3

Iteratively merge two “closest” clusters

Hierarchical Agglomerative Clustering

3

Iteratively merge two “closest” clusters

Hierarchical Agglomerative Clustering

3

Iteratively merge two “closest” clusters

Hierarchical Agglomerative Clustering

3

Iteratively merge two “closest” clusters

Hierarchical Agglomerative Clustering

3

Iteratively merge two “closest” clusters

Hierarchical Agglomerative Clustering

3

Iteratively merge two “closest” clusters

Hierarchical Agglomerative Clustering

3

Iteratively merge two “closest” clusters

4

Inter-Cluster distance given by Linkage Function

Linkage Function

4

Single Linkage (Minimum Pairwise Dissimilarity)

Inter-Cluster distance given by Linkage Function

Linkage Function

5

Single Linkage

Average Linkage (Average Pairwise Dissimilarity)

Linkage Function

6

Average Linkage

Complete Linkage (Maximum Pairwise Dissimilarity)

Linkage Function

Single Linkage

7

Complete Linkage

Best linkage function for a dataset is, a priori, unknown

Linkage Function

Average Linkage Single Linkage

In this work:

• 1. Exponential Linkage: Learnable family of

linkage functions

• 2. Training objective to jointly optimize

linkage & dissimilarity function

8

Exponential Linkage

9

Single Linkage Average Linkage Complete Linkage

Exponential Linkage

9

Single Linkage Average Linkage Complete Linkage

Weighted Average with Learnable Parameter

1 Weight

Exponential Linkage

9

Single Linkage Average Linkage Complete Linkage

Weighted Average with Learnable Parameter

1 Weight

J(θ, α) =

n0

X

i=1

X

Cu,v∈P(i)

• max

n 0, Ψα(Cu0,v0) − Ψα(Cu,v)

• Algorithm 1 train ExpLink(X, C?, T, γ1, γ2)

Init: θ, α for t = 1, . . . , T do J 0 T (0)

j

{xj} 8 xj 2 X for round i = 1, . . . , n0 do {T (i)

k }li k HAC-Round({T (i1) k

}li1

k

) {C(i)}li

k {lvsT (i) k }li k

C(i) {C(i)}li

k

P(i) {Cu,v 2 C(i) ⇥ C(i) : Cu 6= Cv} P(i)

+ {Cu,v 2 P(i) : 9C? j s.t. Cu, Cv ⇢ C? j }

P(i)

P(i) \ P(i) +

Cu0,v0 arg minCu,v2P(i)

+ Ψ↵(Cu,v)

for Cu,v 2 P(i)

• do

J J + max n 0, Ψ↵(Cu0,v0) Ψ↵(Cu,v)

• θ θ γ1 @J

@✓

α α γ2 @J

@↵

Experimental Setup

10

UMIST Faces Noun Phrase Coreference Entity Resolution AMINER REXA

Experimental Setup

Four Linkage Functions

10

UMIST Faces

Single Linkage Average Linkage Complete Linkage Exponential Linkage

Noun Phrase Coreference Entity Resolution AMINER REXA

Experimental Setup

Four Linkage Functions Evaluated using Dendrogram Purity Averaged across 50 different train/test/dev splits

10

UMIST Faces

Single Linkage Average Linkage Complete Linkage Exponential Linkage

Noun Phrase Coreference Entity Resolution AMINER REXA

Results

Dataset: Rexa

Dendrogram Purity

75.0 78.8 82.5 86.3 90.0

Linkage Function

Single Average Complete Exponential

89.1 80.1 84.6 81.7

Results

Dataset: Rexa

Dendrogram Purity

75.0 78.8 82.5 86.3 90.0

Linkage Function

Single Average Complete Exponential

89.1 80.1 84.6 81.7

Summary

12

Summary

12

Exponential Linkage: Learnable family of linkage functions

Summary

12

Exponential Linkage: Learnable family of linkage functions Training Objective & Algorithm: Jointly Optimizing Dissimilarity & Linkage Function

J(θ, α) =

X

X

• max

n 0, Ψα(Cu0,v0) − Ψα(Cu,v)

• Algorithm 1 train ExpLink(X, C?, T, γ1, γ2)

Init: θ, α for t = 1, . . . , T do J 0 T (0)

{xj} 8 xj 2 X for round i = 1, . . . , n0 do {T (i)

}li1

) {C(i)}li

C(i) {C(i)}li

P(i) {Cu,v 2 C(i) ⇥ C(i) : Cu 6= Cv} P(i)

P(i)

Cu0,v0 arg minCu,v2P(i)

for Cu,v 2 P(i)

• do

J J + max n 0, Ψ↵(Cu0,v0) Ψ↵(Cu,v)

• θ θ γ1 @J

α α γ2 @J

Summary

12

Exponential Linkage: Learnable family of linkage functions Training Objective & Algorithm: Jointly Optimizing Dissimilarity & Linkage Function Effective Empirical Results

J(θ, α) =

X

X

• max

n 0, Ψα(Cu0,v0) − Ψα(Cu,v)

• Algorithm 1 train ExpLink(X, C?, T, γ1, γ2)

Init: θ, α for t = 1, . . . , T do J 0 T (0)

{xj} 8 xj 2 X for round i = 1, . . . , n0 do {T (i)

}li1

) {C(i)}li

C(i) {C(i)}li

P(i) {Cu,v 2 C(i) ⇥ C(i) : Cu 6= Cv} P(i)

P(i)

Cu0,v0 arg minCu,v2P(i)

for Cu,v 2 P(i)

• do

J J + max n 0, Ψ↵(Cu0,v0) Ψ↵(Cu,v)

• θ θ γ1 @J

α α γ2 @J

Single Linkage Average Linkage Complete Linkage Exponential Linkage Train Test

Thanks for listening!

Check out our poster #196 today at 6:30pm in Pacific Ballroom!

13

Paper: Code: