Classification from Pairwise Similarity and Unlabeled Data Han Bao - - PowerPoint PPT Presentation

Classification from Pairwise Similarity and Unlabeled Data

Han Bao1,2, Gang Niu2, Masashi Sugiyama2,1

1The University of Tokyo, Japan / 2RIKEN, Japan

July 13th, 2018

Gentle Start: Binary Classification

Training data
 where data is labeled as Goal: find a classifier Method: minimize classification error empirical risk minimization (ERM)

2

Positive data

y = +1

Negative data

y = −1

Boundary

f(x) = 0

yi ∈ {+1, −1}

f : Rd → R

{(xi, yi)}n

i=1 i.i.d.

∼ p(x, y)

xi ∈ Rd

Classification of sensitive matters e.g., politics, religion, opinion on racial issue hard to obtain explicit label instead asking “Which person do you share the same belief as?”

• cf. randomized response technique

3

http://leanintokyo.org/wp-content/uploads/2017/12/MeToo.jpg

two people share the same property

[Warner 1965]

Warner, S. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association, 60(309):63‒69, 1965.

Motivation: Pairwise Information in Classification

Related: Semi-supervised Clustering

Clustering from unlabeled similar dissimilar Offspring of unsupervised clustering Problem: Cluster assumption
 does not hold for many datasets

4

S = {(xi, x0

i)}

D = {(xi, x0

i)}

U = {xi}

[Wagstaff+ ICML2001; many other papers]

different class same class

(manifold assumption, low-density separation)

Wagstaff, K., Cardie, C., Rogers, S., and Schrödl, S. Constrained k-means clustering with background knowledge. In ICML, pp. 577‒584, 2001.

Our work: SU Classification

5

Binary classification from S(imilar) data U(nlabeled) data Formulation based on classification error is available Situation: prediction of sensitive matters e.g., politics, religion “Which person do you share the same belief as?”

S = {(xi, x0

i)}

U = {xi}

S-links belong to the same class

Summary

6

Supervised Classification Unsupervised Clustering

Semi-supervised Clustering similar dissimilar positive negative unlabeled

SU Classification

utilize pairwise information no cluster assumption

Empirical Risk Minimization

Goal: Minimize classification risk (= error rate)
 
 Classification risk
 
 
 Empirical risk
 Margin loss : give small/large penalty for positive/negative value of y f(x) sign(y) = sign(f(x)) ⇔ correct sign(y) ≠ sign(f(x)) ⇔ incorrect 7

expectation over p(x, y)

RPN(f) = E[`(yf(x))]

b RPN(f) = 1 n

n

X

i=1

`(yif(xi))

`

example of loss functions

Assumption: Data Generating Process

Pairwise Similarity (S): belong to the same class
 
 
 Unlabeled Data (U)

8

{xU,i}nU

i=1 ∼ p(x)

π+ , p(y = +1)

π− , p(y = −1)

p+(x) , p(x|y = +1)

p−(x) , p(x|y = −1)

conditional density class prior

pS(x, x0) = p(x, x0 | y = y0 = +1 ∨ y = y0 = −1)

= π2

+p+(x)p+(x0) + π2 p(x)p(x0)

π2

+ + π2

• <latexit sha1_base64="Hc/06EtC8DbDaifh3L3cVL6WAOU=">ADLHichVFLS8NAEB7jq9ZHq14EL8Xi9KyLYIPEIpePNpqVbQ1JHGri3mRpMUa+gf8Ax48KXgQD/4FwYt/wIOIHgXxoKDgxYOTNLWotG7Y3dlvm9mMiPqMjMtQm6buOaW1rZ2X4e/s6u7JxDs7VsxtYIh0YykyZqxJgomlZlKMxazZLqmG1RQRJmuirvzjn+1SA2TaeqyVdJpThG2VZnkmAhxAfXZ7N5Q5DsrM74yGZC5yNjWVGx98rjNXN0PBQJOYSoQ4jWCNFvQrkaoMYs8EwiRF3hf4ac8Ig7cWteADZGELNJCgApQUMFCWwYBTPw2IA4EdMRyYCNmoMVcP4Uy+FbQBZFhoDoLp7b+NrwUBXfTkzTVUuYRcZtoDIEw+SGnJE3ck3OyRP5rBvLdmM4tZTwFitaqvOBg4Glj39VCt4W7NRUDRVroHePO76/2djHco/sYMp0MWZplyO8OwU7qLOD2TKtUW9w/flmbSw/YIOSHP2K1jckusF9q8V06TdH0Efhx3PHfw/1rZBKx6Vg8NRFOznlz98EgDMEYDncSkrAi5DBtJfwC/wyl1wN9wd1+hck2eph9+LO7xC7mxwk8=</latexit><latexit sha1_base64="Hc/06EtC8DbDaifh3L3cVL6WAOU=">ADLHichVFLS8NAEB7jq9ZHq14EL8Xi9KyLYIPEIpePNpqVbQ1JHGri3mRpMUa+gf8Ax48KXgQD/4FwYt/wIOIHgXxoKDgxYOTNLWotG7Y3dlvm9mMiPqMjMtQm6buOaW1rZ2X4e/s6u7JxDs7VsxtYIh0YykyZqxJgomlZlKMxazZLqmG1RQRJmuirvzjn+1SA2TaeqyVdJpThG2VZnkmAhxAfXZ7N5Q5DsrM74yGZC5yNjWVGx98rjNXN0PBQJOYSoQ4jWCNFvQrkaoMYs8EwiRF3hf4ac8Ig7cWteADZGELNJCgApQUMFCWwYBTPw2IA4EdMRyYCNmoMVcP4Uy+FbQBZFhoDoLp7b+NrwUBXfTkzTVUuYRcZtoDIEw+SGnJE3ck3OyRP5rBvLdmM4tZTwFitaqvOBg4Glj39VCt4W7NRUDRVroHePO76/2djHco/sYMp0MWZplyO8OwU7qLOD2TKtUW9w/flmbSw/YIOSHP2K1jckusF9q8V06TdH0Efhx3PHfw/1rZBKx6Vg8NRFOznlz98EgDMEYDncSkrAi5DBtJfwC/wyl1wN9wd1+hck2eph9+LO7xC7mxwk8=</latexit><latexit sha1_base64="Hc/06EtC8DbDaifh3L3cVL6WAOU=">ADLHichVFLS8NAEB7jq9ZHq14EL8Xi9KyLYIPEIpePNpqVbQ1JHGri3mRpMUa+gf8Ax48KXgQD/4FwYt/wIOIHgXxoKDgxYOTNLWotG7Y3dlvm9mMiPqMjMtQm6buOaW1rZ2X4e/s6u7JxDs7VsxtYIh0YykyZqxJgomlZlKMxazZLqmG1RQRJmuirvzjn+1SA2TaeqyVdJpThG2VZnkmAhxAfXZ7N5Q5DsrM74yGZC5yNjWVGx98rjNXN0PBQJOYSoQ4jWCNFvQrkaoMYs8EwiRF3hf4ac8Ig7cWteADZGELNJCgApQUMFCWwYBTPw2IA4EdMRyYCNmoMVcP4Uy+FbQBZFhoDoLp7b+NrwUBXfTkzTVUuYRcZtoDIEw+SGnJE3ck3OyRP5rBvLdmM4tZTwFitaqvOBg4Glj39VCt4W7NRUDRVroHePO76/2djHco/sYMp0MWZplyO8OwU7qLOD2TKtUW9w/flmbSw/YIOSHP2K1jckusF9q8V06TdH0Efhx3PHfw/1rZBKx6Vg8NRFOznlz98EgDMEYDncSkrAi5DBtJfwC/wyl1wN9wd1+hck2eph9+LO7xC7mxwk8=</latexit><latexit sha1_base64="Hc/06EtC8DbDaifh3L3cVL6WAOU=">ADLHichVFLS8NAEB7jq9ZHq14EL8Xi9KyLYIPEIpePNpqVbQ1JHGri3mRpMUa+gf8Ax48KXgQD/4FwYt/wIOIHgXxoKDgxYOTNLWotG7Y3dlvm9mMiPqMjMtQm6buOaW1rZ2X4e/s6u7JxDs7VsxtYIh0YykyZqxJgomlZlKMxazZLqmG1RQRJmuirvzjn+1SA2TaeqyVdJpThG2VZnkmAhxAfXZ7N5Q5DsrM74yGZC5yNjWVGx98rjNXN0PBQJOYSoQ4jWCNFvQrkaoMYs8EwiRF3hf4ac8Ig7cWteADZGELNJCgApQUMFCWwYBTPw2IA4EdMRyYCNmoMVcP4Uy+FbQBZFhoDoLp7b+NrwUBXfTkzTVUuYRcZtoDIEw+SGnJE3ck3OyRP5rBvLdmM4tZTwFitaqvOBg4Glj39VCt4W7NRUDRVroHePO76/2djHco/sYMp0MWZplyO8OwU7qLOD2TKtUW9w/flmbSw/YIOSHP2K1jckusF9q8V06TdH0Efhx3PHfw/1rZBKx6Vg8NRFOznlz98EgDMEYDncSkrAi5DBtJfwC/wyl1wN9wd1+hck2eph9+LO7xC7mxwk8=</latexit>

{(xS,i, x0

S,i)}nS i=1 ∼

Theorem:
 
 
 
 
 is unbiased to
 minimize ⇒ minimize classification risk can be computed only from SU data

9

RPN,`(f) = E [`(yf(x))]

b RSU,`(f) = πS nS

nS

X

i=1

LS,`(f(xS,i)) + LS,`(f(x0

S,i))

2 + 1 nU

nU

X

i=1

LU,`(f(xU,i))

risk for S data risk for U data

(original classification risk)

LS,`(z) , `(z) − `(−z) 2⇡+ − 1

LU,`(z) , −⇡−`(z) + ⇡+`(−z) 2⇡+ − 1

πS , π2

+ + π2 −

π+ , p(y = +1)

π− , p(y = −1)

: surrogate loss

`

b RSU,`(f)

no explicit labels needed

N.B.: πS can be estimated

b RSU,`(f)

Classification risk can be estimated from SU data

Theorem: If the loss function satisfies
 ,
 then the objective (w/ model and l2-reguralization) is convex. Example: squared loss, double-hinge loss, logistic loss

10

`

`(z) − `(−z) = −z

squared loss ⇒ analytical solution (linear system) double-hinge loss ⇒ quadratic program

f(x) = w>φ(x)

Good loss function gives convex objective

computationally efficient to obtain solution

Estimation Error Bound

consistency: estimation error → 0 as n → ∞

• ptimal parametric convergence rate

11

[Mendelson 2008]

Mendelson, S. Lower bounds for the empirical minimization algorithm. IEEE Transactions on Information Theory, 54 (8):3797‒3803, 2008.

RPN( ˆ f) − RPN(f ∗) = Op ✓ 1 √2nS + 1 √nU ◆

# S data # U data estimation error of risk


• f empirical minimizer

RPN(f) = E[`(yf(x))]

f ∗ = argmin

f

RPN(f)

ˆ f = argmin

f

b RSU(f)

true minimizer empirical minimizer

Experiments (Benchmark Datasets)

linear-in-input model / nU = nS = 500 / l2-reguralization baseline: unsupervised / semi-supervised clustering

12

classification accuracies are shown

Summary

Motivation
 
 
 
 
 
 
 SU classification does not need explicit labels Properties: convexity, estimation error bound

13

utilize pairwise information no cluster assumption

Poster: #67

arXiv URL