Conditional tSNE Bo Kang Jefrey Lijffijt Tijl De Bie Ghent - - PowerPoint PPT Presentation

Conditional tSNE

Bo Kang Jefrey Lijffijt Tijl De Bie Darío García García Raúl Santos-Rodríguez

Ghent University Facebook University of Bristol

• High dim data embeddings often have confounding

factors.

• Factors: known, unknown but dominant, sensitive.

Motivation

Motivation

UCI Adult dataset

• High dim data embeddings often has confounding factors
• Factors that are known, unknown but dominant,

sensitive.

• Find embedding with confounding factors explained

away?

Motivation

Method

tSNE

• High dim proximity
• Low dim proximity
• Objective:

min

y KL(P||Q)

H(P)

H(Q)

H(P, Q)

KL(P||Q)

P

Q

• Confounding factors are encoded by labels,

denoted by delta matrix

• Formalize lower dim distribution
• Objective:

Method

ctSNE

min

y KL(P||R)

∆

H(Q|∆)

H(P)

H(Q)

H(∆)

KL(P||R)

rij = P(i, j|∆)

Method

A bit of Bayesian

P(∆|i, j) = αδijβ1−δij

P(∆) = X

i6=j

P(∆|i, j)P(i, j)

P(i, j) = qij

P(i, j|∆) = P(∆|i, j)P(i, j) P(∆) = αδijβδijqij α P

i6=j:δij=1 qij + β P i6=j:δij=0 qij

Method

Intuition tSNE

∂ ∂yi KL(P||Q) ∝ X

j6=i

pijqij(yi − yj) − X

j6=i

q2

ij(yi − yj)

Repelling force Attraction force

Method

Intuition ctSNE

∂ ∂yi KL(P||R) ∝ X

j6=i

pijqij(yi − yj) − δijα0 + (1 − δij)β0 O X

j6=i

q2

ij(yi − yj)

O = α0 X

i6=j:δij=1

qij + β0 X

i6=j:δij=0

qij

where

Weighted repelling force Attraction force

1 = α0 P

l nl(nl − 1)

n(n − 1) + β0 ✓ 1 − P

l nl(nl − 1)

n(n − 1) ◆

α0 , α n! Q

l nl!, β0 , β

n! Q

l nl!

1 = X

∆

P(∆|i, j) = X

∆

αδijβ1−δij

where

Method

Intuition ctSNE ranging alpha

• Label information

allows to achieve a closer distribution.

• Exact:
• High dim: a vantage-point tree
• Low dim: quad tree
• ctSNE implementation: quad tree with label count.

Algorithm

Barnes-Hut approximation

O(n2)

O(n log n)

O(n log n)

Experiments

Case study: synthetic dataset

Experiments

Case study: synthetic dataset

Experiments

Case study: synthetic dataset

Experiments

Case study: synthetic dataset

Experiments

Case study: UCI Adult dataset

• 15
• 10
• 5

5 10 15 20

• 20
• 15
• 10
• 5

5 10 15

tSNE

female male

• 15
• 10
• 5

5 10 15

• 15
• 10
• 5

5 10 15 20

ctSNE with gender prior alphaPrime = 1.80, betaPrime = -0.22

female male

Experiments

Case study: UCI Adult dataset

• 15
• 10
• 5

5 10 15 20

• 20
• 15
• 10
• 5

5 10 15

tSNE

asian black

• ther

white

• 15
• 10
• 5

5 10 15 20

• 10
• 5

5 10 15

ctSNE with ethnic group prior alphaPrime = 1.30, betaPrime = -0.12

asian black

• ther

white

low income, white female high income, white female low income, white male; plus all other ethnic groups high income, white male white, low income white, high income

• 15
• 10
• 5

5 10 15 20

• 20
• 15
• 10
• 5

5 10 15

tSNE

asian black

• ther

white

• 20
• 15
• 10
• 5

5 10 15

• 15
• 10
• 5

5 10 15 20 25

ctSNE with ethnic group and gender prior alphaPrime = 2.10, betaPrime = -0.10

asian black

• ther

white

high income low income; plus all other ethnic groups

Experiments

Case study: UCI Adult dataset

• Show me some figures

Experiments

Case study: Facebook dataset

• Construct KNN graph on embedding
• Measure the consistency of a factor over the graph
• Normalized Laplacian score:
• Lager the score, less consistent over the graph

Experiments

Quantitative benchmark

f 0D1/2LD1/2f

Experiments

Quantitative benchmark

Experiments

Quantitative benchmark

Experiments

Quantitative benchmark

• Remove label information then apply tSNE
• Example:

Adversarial Auto Encoder then tSNE

Experiments

Baseline

Experiments

Baseline: AAE + tSNE

Experiments

Baseline: AAE + tSNE

Experiments

Parameter sensitivity GIF

Experiments

Parameter sensitivity GIF

Experiments

Parameter sensitivity GIF

• 100k x 128d data
• 55GiB RAM / 1GiB SWAP
• 24 x 2.494 GHz
• 9600s without approximation
• 9s per gradient update with tree approximation
• 52s per update with 500k x 128d data

Experiments

Runtime

• Interpret cluster (in)consistent between tSNE and cTSNE
• Automatic cluster/region finding
• Histogram interpretation
• Logistic regression on sparse feature (e.g., links, tags)
• Random Forest on sparse feature
• Embedding space KNN
• Subgroup discovery based methods

Experiments

Interpretation methods

• Generalized framework
• Independence assumption
• Steerable / iterative data exploration: align or explain

away

• Privacy preserving data analysis
• Optimization

Future Work

Thanks