SVMpAUC-tight: A new algorithm for optimizing partial AUC based on a - - PowerPoint PPT Presentation

▶
svmpauc tight a new algorithm for optimizing partial auc
SMART_READER_LITE
LIVE PREVIEW

SVMpAUC-tight: A new algorithm for optimizing partial AUC based on a - - PowerPoint PPT Presentation

Dec 14, 2023 218 likes •1.13k views

SVMpAUC-tight: A new algorithm for optimizing partial AUC based on a tight convex upper bound Harikrishna Narasimhan and Shivani Agarwal Department of Computer Science and Automation Indian Institute of Science, Bangalore Receiver Operating

slide-1
SLIDE 1

SVMpAUC-tight: A new algorithm for

  • ptimizing partial AUC based on a

tight convex upper bound

Harikrishna Narasimhan and Shivani Agarwal

Department of Computer Science and Automation Indian Institute of Science, Bangalore

slide-2
SLIDE 2

Receiver Operating Characteristic Curve

slide-3
SLIDE 3

Binary Classification

Vs.

Non-Spam Spam

Area Under the ROC Curve (AUC)

Receiver Operating Characteristic Curve

slide-4
SLIDE 4

Binary Classification

Vs.

Non-Spam Spam

Bipartite Ranking

Ranking of documents Area Under the ROC Curve (AUC)

Receiver Operating Characteristic Curve

slide-5
SLIDE 5

Partial AUC?

Full AUC

slide-6
SLIDE 6

Partial AUC?

Vs Full AUC Partial AUC

slide-7
SLIDE 7

Ranking

http://www.google.com/

slide-8
SLIDE 8

Ranking

http://www.google.com/

slide-9
SLIDE 9

Medical Diagnosis

http://en.wikipedia.org/

slide-10
SLIDE 10

Medical Diagnosis

KDD Cup 2008

http://en.wikipedia.org/

slide-11
SLIDE 11

Bioinformatics

― Drug Discovery ― Gene Prioritization ― Protein Interaction Prediction ― ……

http://en.wikipedia.org/wiki http://commons.wikimedia.org/ http://www.google.com/imghp

slide-12
SLIDE 12

Bioinformatics

― Drug Discovery ― Gene Prioritization ― Protein Interaction Prediction ― ……

http://en.wikipedia.org/wiki http://commons.wikimedia.org/ http://www.google.com/imghp

slide-13
SLIDE 13

Partial Area Under the ROC Curve is critical to many applications

slide-14
SLIDE 14

Narasimhan, H. and Agarwal, S. “A structural SVM based approach for optimizing partial AUC”, ICML 2013.

SVMpAUC (ICML 2013)

SVMpAUC

slide-15
SLIDE 15

Narasimhan, H. and Agarwal, S. “A structural SVM based approach for optimizing partial AUC”, ICML 2013.

SVMpAUC (ICML 2013)

SVMpAUC SVM-AUC Joachims, 2005

slide-16
SLIDE 16

Improved Version of SVMpAUC

Tighter upper bound Improved accuracy Better runtime guarantee

slide-17
SLIDE 17

Outline

  • Overview of SVMpAUC
  • Upper Bound Optimized by SVMpAUC
  • Improved Formulation: SVMpAUC-tight
  • Optimization Methods
  • Experiments
slide-18
SLIDE 18

Receiver Operating Characteristic Curve

Positive Instances Negative Instances ……..

x1

+

x2

+

x3

+

xm

+

……..

x1

  • x2
  • x3
  • xn
  • Training

Set

Learn a scoring function GOAL?

slide-19
SLIDE 19

Receiver Operating Characteristic Curve

Positive Instances Negative Instances ……..

x1

+

x2

+

x3

+

xm

+

……..

x1

  • x2
  • x3
  • xn
  • Training

Set

Learn a scoring function GOAL?

Rank objects

….

x3

+

x5

+

x6

+

x1

  • xn
  • Build a classifier

….

x3

+

x5

+

x6

+

x1

  • xn
  • r

Threshold

slide-20
SLIDE 20

Receiver Operating Characteristic Curve

Positive Instances Negative Instances ……..

x1

+

x2

+

x3

+

xm

+

……..

x1

  • x2
  • x3
  • xn
  • Training

Set

Learn a scoring function GOAL?

Rank objects

….

x3

+

x5

+

x6

+

x1

  • xn
  • Build a classifier

….

x3

+

x5

+

x6

+

x1

  • xn
  • r

Threshold Quality of scoring function?

Threshold Assignment

slide-21
SLIDE 21

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives

slide-22
SLIDE 22

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives

slide-23
SLIDE 23

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives

slide-24
SLIDE 24

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives

slide-25
SLIDE 25

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives Area Under the ROC Curve (AUC)

slide-26
SLIDE 26

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives Area Under the ROC Curve (AUC) Partial AUC

slide-27
SLIDE 27

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives

slide-28
SLIDE 28

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives

slide-29
SLIDE 29

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives

slide-30
SLIDE 30

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives

slide-31
SLIDE 31

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives

β = 0.5

Top 3 negatives!

slide-32
SLIDE 32

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives

β = 0.5

Top 3 negatives!

slide-33
SLIDE 33

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives

β = 0.5

Top 3 negatives!

slide-34
SLIDE 34

20 15 14 13 11 9 8 6 5 3 2

Receiver Operating Characteristic Curve ROC Curve

False Positives True Positives

β = 0.5

Top 3 negatives!

slide-35
SLIDE 35

(1 – pAUC) for f

slide-36
SLIDE 36

(1 – pAUC) for f

Convex Upper Bound

slide-37
SLIDE 37

(1 – pAUC) for f

Convex Upper Bound

+ Regularizer

slide-38
SLIDE 38

SVMpAUC (ICML 2013)

SVMpAUC: Structural SVM Approach Narasimhan and Agarwal, 2013

slide-39
SLIDE 39

SVMpAUC (ICML 2013)

Ordering of training examples:

1 1 1 1 1 1 1 1 m n

SVMpAUC: Structural SVM Approach Narasimhan and Agarwal, 2013

slide-40
SLIDE 40

SVMpAUC (ICML 2013)

Ordering of training examples:

1 1 1 1 1 1 1 1 m n

SVMpAUC: Structural SVM Approach Narasimhan and Agarwal, 2013

Scoring function f

slide-41
SLIDE 41

SVMpAUC (ICML 2013)

Ordering of training examples:

1 1 1 1 1 1 1 1 m n

SVMpAUC: Structural SVM Approach Narasimhan and Agarwal, 2013

Scoring function f

slide-42
SLIDE 42

SVMpAUC (ICML 2013)

Ordering of training examples:

1 1 1 1 1 1 1 1 m n

SVMpAUC: Structural SVM Approach Narasimhan and Agarwal, 2013

Scoring function f

slide-43
SLIDE 43

(1 – pAUC) for f

Convex Upper Bound

+ Regularizer

≤

slide-44
SLIDE 44

(1 – pAUC) for f

Convex Upper Bound

+ Regularizer

≤

How does this upper bound look?

slide-45
SLIDE 45

(1 – pAUC) for f

Convex Upper Bound

+ Regularizer

≤

Can we obtain a tighter upper bound?

slide-46
SLIDE 46

Outline

  • Overview of SVMpAUC
  • Upper Bound Optimized by SVMpAUC
  • Improved Formulation: SVMpAUC-tight
  • Optimization Methods
  • Experiments
slide-47
SLIDE 47

1 - pAUC Upper bound we want? ∝

slide-48
SLIDE 48

1 - pAUC Upper bound we want? ∝

slide-49
SLIDE 49

1 - pAUC Upper bound we want? ∝

slide-50
SLIDE 50

1 - pAUC Upper bound we want? ≤ ∝

pair-wise hinge loss!

slide-51
SLIDE 51

Upper optimized by SVMpAUC?

slide-52
SLIDE 52

= pair-wise hinge loss + extra term Upper optimized by SVMpAUC?

slide-53
SLIDE 53

= pair-wise hinge loss + extra term Upper optimized by SVMpAUC?

Subset of pairs of positive-negative examples

slide-54
SLIDE 54

= pair-wise hinge loss + extra term Upper optimized by SVMpAUC?

?

Subset of pairs of positive-negative examples

slide-55
SLIDE 55

Upper optimized by SVMpAUC?

slide-56
SLIDE 56

Upper optimized by SVMpAUC? ≤ pair-wise hinge loss + extra term

slide-57
SLIDE 57

Upper optimized by SVMpAUC? ≤ pair-wise hinge loss + extra term

  • approx. pair-wise hinge loss + extra term

≤

slide-58
SLIDE 58

Upper optimized by SVMpAUC? ≤ pair-wise hinge loss + extra term

  • approx. pair-wise hinge loss + extra term

≤

? ?

slide-59
SLIDE 59

Outline

  • Overview of SVMpAUC
  • Upper Bound Optimized by SVMpAUC
  • Improved Formulation: SVMpAUC-tight
  • Optimization Methods
  • Experiments
slide-60
SLIDE 60

20 15 14 13 11 9 8 6 5 3 2

Rewriting the Partial AUC Loss

False Positives True Positives

3 + 2 + 2 = 7 α = 0, β = 0.5

slide-61
SLIDE 61

20 15 14 13 11 9 8 6 5 3 2

Rewriting the Partial AUC Loss

False Positives True Positives

3 + 2 + 2 = 7 2 + 2 + 1 = 5 α = 0, β = 0.5

slide-62
SLIDE 62

20 15 14 13 11 9 8 6 5 3 2

Rewriting the Partial AUC Loss

False Positives True Positives

3 + 2 + 2 = 7 2 + 2 + 1 = 5 . . . 1 + 1 + 1 = 3 α = 0, β = 0.5

slide-63
SLIDE 63

20 15 14 13 11 9 8 6 5 3 2

Rewriting the Partial AUC Loss

False Positives True Positives

3 + 2 + 2 = 7 2 + 2 + 1 = 5 . . . 1 + 1 + 1 = 3

1 - AUC restricted to top β fraction of negatives

Maximum!

α = 0, β = 0.5

slide-64
SLIDE 64

Top jβ negatives

slide-65
SLIDE 65

SVM-AUC Top jβ negatives

slide-66
SLIDE 66

Negatives jα to jβ

slide-67
SLIDE 67

Negatives jα to jβ Truncated SVMpAUC

slide-68
SLIDE 68

SVMpAUC-tight: Improved Formulation

SVMpAUC objective restricted to S

slide-69
SLIDE 69

SVMpAUC-tight: Improved Formulation

Top jβ negatives

slide-70
SLIDE 70

SVMpAUC-tight: Improved Formulation

Same pairs of positive-negative examples

= pair-wise hinge loss + extra term

Top jβ negatives

slide-71
SLIDE 71

SVMpAUC-tight: Improved Formulation

Negatives jα to jβ

slide-72
SLIDE 72

SVMpAUC-tight: Improved Formulation

≤ pair-wise hinge loss + extra term

  • approx. pair-wise hinge loss + extra term

≤

Negatives jα to jβ

slide-73
SLIDE 73

Outline

  • Overview of SVMpAUC
  • Upper Bound Optimized by SVMpAUC
  • Improved Formulation: SVMpAUC-tight
  • Optimization Methods
  • Experiments
slide-74
SLIDE 74

SVMpAUC-tight: Optimization Problem

+ Regularizer

exponential in size

slide-75
SLIDE 75

SVMpAUC-tight: Optimization Problem

+ Regularizer

exponential in size

Quadratic program with an exponential number of constraints

slide-76
SLIDE 76

SVMpAUC-tight: Cutting-Plane Solver

Repeat: 1. Solve OP for a subset of constraints. 2. Add the most violated constraint.

slide-77
SLIDE 77

SVMpAUC-tight: Cutting-Plane Solver

Repeat: 1. Solve OP for a subset of constraints. 2. Add the most violated constraint.

slide-78
SLIDE 78

SVMpAUC-tight: Cutting-Plane Solver

Repeat: 1. Solve OP for a subset of constraints. 2. Add the most violated constraint.

Better Runtime Guarantees: Maximum number of iterations Time taken per iteration

slide-79
SLIDE 79

SVMpAUC-tight: Projected Subgradient Solver

Primal formulation:

slide-80
SLIDE 80

SVMpAUC-tight: Projected Subgradient Solver

Repeat: 1. Compute subgradient and perform update 2. Project on to the constraint set.

Primal formulation:

slide-81
SLIDE 81

SVMpAUC-tight: Projected Subgradient Solver

Repeat: 1. Compute subgradient and perform update 2. Project on to the constraint set.

Primal formulation:

Sparsity-inducing regularizations

LASSO Group LASSO Elastic-Net

slide-82
SLIDE 82

Outline

  • Overview of SVMpAUC
  • Upper Bound Optimized by SVMpAUC
  • Improved Formulation: SVMpAUC-tight
  • Optimization Methods
  • Experiments
slide-83
SLIDE 83

Partial AUC in [0, 0.1] Partial AUC in [0.2s, 0.3s]

SVMpAUC-tight Vs SVMpAUC

Leukemia PPI Chem- informatics KDD Cup 2001 Ovarian Cancer SVMpAUC-tight 30.44 52.95 65.30 69.91 91.84 SVMpAUC 24.64 51.96 65.28 70.12 91.84 SVMAUC 28.83 39.72 62.78 62.23 92.17 KDD Cup 2008 SVMpAUC-tight 53.43 SVMpAUC 51.89 SVMAUC 50.66

slide-84
SLIDE 84

Run-time Analysis

Interval [0, β]

Repeat: 1. Solve OP for a subset of constraints. 2. Add the most violated constraint.

slide-85
SLIDE 85

Run-time Analysis

Interval [0, β]

Repeat: 1. Solve OP for a subset of constraints. 2. Add the most violated constraint.

slide-86
SLIDE 86

Run-time Analysis

Interval [0, β]

Repeat: 1. Solve OP for a subset of constraints. 2. Add the most violated constraint.

slide-87
SLIDE 87

Cutting-Plane vs. Projected Subgradient

Cutting-plane method is faster on high dimensional data with L2 regularization Projected subgradient method is faster with L1 regularization

slide-88
SLIDE 88

Sparse and Group Sparse Extensions

Sparse models at the cost of decrease in accuracy

slide-89
SLIDE 89

Conclusions

  • A new support vector algorithm for optimizing

partial AUC based on a tight convex upper bound

  • Cutting-plane solver with better run-time

guarantees

  • Experiments on several bioinformatics tasks

demonstrate improved accuracy

  • Projected subgradient solver allows sparse and

group sparse extensions

slide-90
SLIDE 90

Questions?