New Regularized Algorithms for Transductive Learning Partha Pratim - - PowerPoint PPT Presentation

New Regularized Algorithms for Transductive Learning

Partha Pratim Talukdar University of Pennsylvania, USA Koby Crammer Technion, Israel

1

Graph-based Semi-Supervised Learning

2

0.3 0.2 0.2 0.1 0.2 0.3 0.2

Graph-based Semi-Supervised Learning

2

Labeled (seed)

0.3 0.2 0.2 0.1 0.2 0.3 0.2

Graph-based Semi-Supervised Learning

2

Unlabeled Labeled (seed)

0.3 0.2 0.2 0.1 0.2 0.3 0.2

Graph-based Semi-Supervised Learning

3

0.3 0.2 0.2 0.1 0.2 0.3 0.2

Graph-based Semi-Supervised Learning

3

Various methods: LP (Zhu et al., 2003); QC (Bengio et al., 2007); Adsorption (Baluja et al., 2008)

0.3 0.2 0.2 0.1 0.2 0.3 0.2

Adsorption Algorithm

4

Adsorption Algorithm

• Successfully used in
• YouTube

Video Recommendation [Baluja et al., 2008], Semantic Classification [Talukdar et al., 2008]

4

Adsorption Algorithm

• Successfully used in
• YouTube

Video Recommendation [Baluja et al., 2008], Semantic Classification [Talukdar et al., 2008]

• It has not been analyzed so far
• Is it optimizing an objective? If so, what?
• Motivation for proposed work

4

Adsorption Algorithm

[Baluja et al., WWW 2008]

0.3 0.3 0.2

Adsorption Algorithm

[Baluja et al., WWW 2008]

0.3 0.3 0.2

Adsorption Algorithm

[Baluja et al., WWW 2008]

0.3 0.3 0.2 Dummy Label

Characteristics of Adsorption

6

Characteristics of Adsorption

• Highly scalable and iterative

6

Characteristics of Adsorption

• Highly scalable and iterative
• Main difference with previous methods:
• all nodes are not equal: high-degree nodes

are discounted

6

Characteristics of Adsorption

• Highly scalable and iterative
• Main difference with previous methods:
• all nodes are not equal: high-degree nodes

are discounted

• Two equivalent views:

6

Characteristics of Adsorption

• Highly scalable and iterative
• Main difference with previous methods:
• all nodes are not equal: high-degree nodes

are discounted

• Two equivalent views:

6

U L Label Diffusion L Random Walk U

Random Walk View

7

U V

Random Walk View

7

U V

what next?

Random Walk View

7

U V

what next?

• Continue walk with prob.
• Assign

V’s seed label to U with prob.

• Abandon random walk with prob.
• assign U a dummy label with score

pcont

v

pinj

v

pabnd

v

pabnd

v

Discounting High-Degree Nodes

Discounting High-Degree Nodes

• High-degree nodes can be unreliable
• do not allow propagation/walk through

them

Discounting High-Degree Nodes

• High-degree nodes can be unreliable
• do not allow propagation/walk through

them

• Solution: increase abandon probability on high-

degree nodes

Discounting High-Degree Nodes

• High-degree nodes can be unreliable
• do not allow propagation/walk through

them

• Solution: increase abandon probability on high-

degree nodes

pabnd

v

∝ degree(v)

Is Adsorption Optimizing an Objective?

Is Adsorption Optimizing an Objective?

• Under certain assumptions, NO!
• Theorem in the paper

Is Adsorption Optimizing an Objective?

• Under certain assumptions, NO!
• Theorem in the paper
• Our Goal:
• Retain Adsorption’s desirable properties
• But do so using a well-defined optimization

Is Adsorption Optimizing an Objective?

• Under certain assumptions, NO!
• Theorem in the paper
• Our Goal:
• Retain Adsorption’s desirable properties
• But do so using a well-defined optimization
• Proposed Solution: MAD (next slide)

Modified Adsorption (MAD)

[This Paper]

10

MAD Objective

Modified Adsorption (MAD)

[This Paper]

10

MAD Objective

min{ˆ

yvl}

• l
• µ1
• v

pinj

v (yvl − ˆ

yvl)2 + µ2

• u
• v

w

• uv(ˆ

yul − ˆ yvl)2 + µ3

• v

(rvl − ˆ yvl)2

Modified Adsorption (MAD)

[This Paper]

10

MAD Objective

Smoothness Loss Across Edge Label Prior Loss (e.g. prior on dummy label)

+ +

Seed Label Loss (if any)

min

Modified Adsorption (MAD)

[This Paper]

• High-degree node discounting is enforced through the

third term

10

MAD Objective

Smoothness Loss Across Edge Label Prior Loss (e.g. prior on dummy label)

+ +

Seed Label Loss (if any)

min

Modified Adsorption (MAD)

[This Paper]

• High-degree node discounting is enforced through the

third term

• Results in an Adsorption-like iterative update, scalable

10

MAD Objective

Smoothness Loss Across Edge Label Prior Loss (e.g. prior on dummy label)

+ +

Seed Label Loss (if any)

min

Extension to Dependent Labels

• Labels are not always mutually exclusive.

11

Extension to Dependent Labels

• Labels are not always mutually exclusive.

11

BrownAle 1.0 Ale PaleAle ScotchAle 1.0 1.0 TopFormentedBeer 0.95 White Porter 0.8 0.8

Extension to Dependent Labels

• Labels are not always mutually exclusive.

11

Label Graph

BrownAle 1.0 Ale PaleAle ScotchAle 1.0 1.0 TopFormentedBeer 0.95 White Porter 0.8 0.8

Labels Label Similarity

MAD with Dependent Labels (MADDL)

12

min

Edge Smoothness Loss Label Prior Loss (e.g. prior

• n dummy label)

+ +

Seed Label Loss (if any)

MAD with Dependent Labels (MADDL)

12

min

Edge Smoothness Loss Label Prior Loss (e.g. prior

• n dummy label)

+ +

Seed Label Loss (if any)

+

Dependent Label Loss

MAD with Dependent Labels (MADDL)

12

MADDL Objective min

Edge Smoothness Loss Label Prior Loss (e.g. prior

• n dummy label)

+ +

Seed Label Loss (if any)

+

Dependent Label Loss

MAD with Dependent Labels (MADDL)

12

MADDL Objective min

Edge Smoothness Loss Label Prior Loss (e.g. prior

• n dummy label)

+ +

Seed Label Loss (if any)

+

Dependent Label Loss Penalize if similar labels are assigned different scores on a node

MAD with Dependent Labels (MADDL)

12

MADDL Objective min

Edge Smoothness Loss Label Prior Loss (e.g. prior

• n dummy label)

+ +

Seed Label Loss (if any)

+

Dependent Label Loss Penalize if similar labels are assigned different scores on a node BrownAle 1.0 Ale

MAD with Dependent Labels (MADDL)

12

MADDL Objective

• MADDL objective results in a

scalable iterative update, with convergence guarantee.

min

Edge Smoothness Loss Label Prior Loss (e.g. prior

• n dummy label)

+ +

Seed Label Loss (if any)

+

Dependent Label Loss Penalize if similar labels are assigned different scores on a node BrownAle 1.0 Ale

Experimental Setup

13

Experimental Setup

• I. Classification Experiments
• WebKB (4 classes) [Subramanya and Bilmes, 2008]
• Sentiment Classification (4 classes) [Blitzer,

Dredze and Pereira, 2007]

• k-Nearest Neighbor Graph (k is tuned)

13

Experimental Setup

• I. Classification Experiments
• WebKB (4 classes) [Subramanya and Bilmes, 2008]
• Sentiment Classification (4 classes) [Blitzer,

Dredze and Pereira, 2007]

• k-Nearest Neighbor Graph (k is tuned)
• II. Smoother Sentiment Ranking with MADDL

13

PRBEP (macro-averaged) on WebKB Dataset, 3148 test instances

14

(Zhu et al., 03)

Precision on 3568 Sentiment test instances

(Zhu et al., 03)

• II. Smooth Sentiment Ranking

16

smooth predictions

rank 1 rank 4

• II. Smooth Sentiment Ranking

16

non-smooth predictions smooth predictions

rank 1 rank 4

• II. Smooth Sentiment Ranking

16

• ver

Prefer

non-smooth predictions smooth predictions

rank 1 rank 4

• II. Smooth Sentiment Ranking

16

• ver

Prefer

non-smooth predictions 1.0 1.0

MADDL Label Constraints

smooth predictions

rank 1 rank 4

1 2 3 4 1 2 3 4 5000 10000 Label 1 Count of Top Predicted Pair in MADDL Output Label 2

• II. Smooth Sentiment Ranking

1 2 3 4 1 2 3 4 2000 4000 Label 1 Count of Top Predicted Pair in MAD Output Label 2

1 2 3 4 1 2 3 4 5000 10000 Label 1 Count of Top Predicted Pair in MADDL Output Label 2

• II. Smooth Sentiment Ranking

1 2 3 4 1 2 3 4 2000 4000 Label 1 Count of Top Predicted Pair in MAD Output Label 2

MADDL generates smoother ranking, while preserving accuracy of prediction.

Conclusion

18

Conclusion

• Presented Modified Adsorption (MAD)
• an Adsorption-like algorithm but with well defined
• ptimization.

18

Conclusion

• Presented Modified Adsorption (MAD)
• an Adsorption-like algorithm but with well defined
• ptimization.
• Extended MAD to MADDL
• MADDL can handle non mutually-exclusive labels.

18

Conclusion

• Presented Modified Adsorption (MAD)
• an Adsorption-like algorithm but with well defined
• ptimization.
• Extended MAD to MADDL
• MADDL can handle non mutually-exclusive labels.
• Demonstrated effectiveness of MAD and

MADDL on real world datasets.

18

Conclusion

• Presented Modified Adsorption (MAD)
• an Adsorption-like algorithm but with well defined
• ptimization.
• Extended MAD to MADDL
• MADDL can handle non mutually-exclusive labels.
• Demonstrated effectiveness of MAD and

MADDL on real world datasets.

• Future Work
• Apply MADDL in other domains with dependent

labels e.g. Information Extraction

18

Thanks!

19

algorithm authors