From ranking to intransitive preference learning: - - PowerPoint PPT Presentation

From ranking to intransitive preference learning: rock-paper-scissors and beyond

Tapio Pahikkala1 Willem Waegeman2 Evgeni Tsivtsivadze3 Tapio Salakoski1 Bernard De Baets2

1TUCS, University of Turku, Finland 2KERMIT, Ghent University, Belgium 3Institute for Computing and Information Sciences

Radboud University Nijmegen, The Netherlands

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 1 / 25

Introduction

Outline

1

Introduction

2

Stochastic transitivity and ranking representability

3

Learning intransitive reciprocal relations

4

Experiments

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 2 / 25

Introduction

The transitivity property: a classical example

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 3 / 25

Introduction

Examples of intransitivity are found in many fields...

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 4 / 25

Stochastic transitivity and ranking representability

Outline

1

Introduction

2

Stochastic transitivity and ranking representability

3

Learning intransitive reciprocal relations

4

Experiments

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 5 / 25

Stochastic transitivity and ranking representability

Q(x, x′) = 5/9 Q(x′, x′′) = 5/9 Q(x′′, x) = 5/9 Q(x′, x) = 4/9 Q(x′′, x′) = 4/9 Q(x, x′′) = 4/9 Proposition A relation Q : X 2 → [0, 1] is called a reciprocal relation if Q(x, x′) + Q(x′, x) = 1 ∀(x, x′) ∈ X 2 .

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 6 / 25

Stochastic transitivity and ranking representability

Q(x, x′) = 5/9 Q(x′, x′′) = 5/9 Q(x′′, x) = 5/9 Q(x′, x) = 4/9 Q(x′′, x′) = 4/9 Q(x, x′′) = 4/9 Proposition A relation Q : X 2 → [0, 1] is called a reciprocal relation if Q(x, x′) + Q(x′, x) = 1 ∀(x, x′) ∈ X 2 .

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 6 / 25

Stochastic transitivity and ranking representability

Q(x, x′) = 5/9 Q(x′, x′′) = 5/9 Q(x′′, x) = 5/9 Q(x′, x) = 4/9 Q(x′′, x′) = 4/9 Q(x, x′′) = 4/9 Proposition A relation Q : X 2 → [0, 1] is called a reciprocal relation if Q(x, x′) + Q(x′, x) = 1 ∀(x, x′) ∈ X 2 .

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 6 / 25

Stochastic transitivity and ranking representability

Q(x, x′) = 5/9 Q(x′, x′′) = 5/9 Q(x′′, x) = 5/9 Q(x′, x) = 4/9 Q(x′′, x′) = 4/9 Q(x, x′′) = 4/9 Proposition A relation Q : X 2 → [0, 1] is called a reciprocal relation if Q(x, x′) + Q(x′, x) = 1 ∀(x, x′) ∈ X 2 .

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 6 / 25

Stochastic transitivity and ranking representability

Ranking representability

Definition A reciprocal relation Q : X 2 → [0, 1] is called weakly ranking representable if there exists a ranking function f : X → R such that for any (x, x′) ∈ X 2 it holds that Q(x, x′) ≤ 1 2 ⇔ f(x) ≤ f(x′) . Q(x, x′) = 5/9 ⇔ x ≻ x′ Q(x′, x′′) = 5/9 ⇔ x′ ≻ x′′ Q(x′′, x) = 5/9 ⇔ x′′ ≻ x

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 7 / 25

Stochastic transitivity and ranking representability

Ranking representability

Definition A reciprocal relation Q : X 2 → [0, 1] is called weakly ranking representable if there exists a ranking function f : X → R such that for any (x, x′) ∈ X 2 it holds that Q(x, x′) ≤ 1 2 ⇔ f(x) ≤ f(x′) . Q(x, x′) = 5/9 ⇔ x ≻ x′ Q(x′, x′′) = 5/9 ⇔ x′ ≻ x′′ Q(x′′, x) = 5/9 ⇔ x′′ ≻ x

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 7 / 25

Stochastic transitivity and ranking representability

Weak stochastic transitivity

Proposition (Luce and Suppes, 1965) A reciprocal relation Q is weakly ranking representable if and only if it satisfies weak stochastic transitivity, i.e., for any (x, x′, x′′) ∈ X 3 it holds that Q(x, x′) ≥ 1/2 ∧ Q(x′, x′′) ≥ 1/2 ⇒ Q(x, x′′) ≥ 1/2 . Q(x, x′) = 6/9 Q(x′, x′′) = 5/9 Q(x′′, x) = 2/9 ⇔ x ≻ x′ ≻ x′′

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 8 / 25

Stochastic transitivity and ranking representability

Weak stochastic transitivity

Proposition (Luce and Suppes, 1965) A reciprocal relation Q is weakly ranking representable if and only if it satisfies weak stochastic transitivity, i.e., for any (x, x′, x′′) ∈ X 3 it holds that Q(x, x′) ≥ 1/2 ∧ Q(x′, x′′) ≥ 1/2 ⇒ Q(x, x′′) ≥ 1/2 . Q(x, x′) = 6/9 Q(x′, x′′) = 5/9 Q(x′′, x) = 2/9 ⇔ x ≻ x′ ≻ x′′

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 8 / 25

Learning intransitive reciprocal relations

Outline

1

Introduction

2

Stochastic transitivity and ranking representability

3

Learning intransitive reciprocal relations

4

Experiments

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 9 / 25

Learning intransitive reciprocal relations

Definition of our framework

Training data E = (ei, yi)N

i=1

Training data are here couples: e = (x, x′) Labels yi = 2Q(xi, x′

i) + 1

Minimizing the regularized empirical error: A(E) = argmin

h∈F

1 N

N

• i=1

L(h(ei), yi) + λh2

F

Least-squares loss function: regularized least-squares

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 10 / 25

Learning intransitive reciprocal relations

Reciprocal relations are learned by defining a specific kernel construction

Consider the following joint feature representation for a couple: Φ(ei) = Φ(xi, x′

i) = Ψ(xi, x′ i) − Ψ(x′ i, xi),

This yields the following kernel defined on couples: K Φ(ei, ej) = K Φ(xi, x′

i, xj, x′ j)

= Ψ(xi, x′

i) − Ψ(x′ i, xi), Ψ(xj, x′ j) − Ψ(x′ j, xj)

= K Ψ(xi, x′

i, xj, x′ j) + K Ψ(x′ i, xi, x′ j, xj)

−K Ψ(x′

i, xi, xj, x′ j) − K Ψ(xi, x′ i, x′ j, xj) .

And the model becomes: h(x, x′) = w, Ψ(x, x′) − Ψ(x′, x) =

N

• i=1

aiK Φ(xi, x′

i, x, x′) .

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 11 / 25

Learning intransitive reciprocal relations

Reciprocal relations are learned by defining a specific kernel construction

Consider the following joint feature representation for a couple: Φ(ei) = Φ(xi, x′

i) = Ψ(xi, x′ i) − Ψ(x′ i, xi),

This yields the following kernel defined on couples: K Φ(ei, ej) = K Φ(xi, x′

i, xj, x′ j)

= Ψ(xi, x′

i) − Ψ(x′ i, xi), Ψ(xj, x′ j) − Ψ(x′ j, xj)

= K Ψ(xi, x′

i, xj, x′ j) + K Ψ(x′ i, xi, x′ j, xj)

−K Ψ(x′

i, xi, xj, x′ j) − K Ψ(xi, x′ i, x′ j, xj) .

And the model becomes: h(x, x′) = w, Ψ(x, x′) − Ψ(x′, x) =

N

• i=1

aiK Φ(xi, x′

i, x, x′) .

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 11 / 25

Learning intransitive reciprocal relations

Reciprocal relations are learned by defining a specific kernel construction

Consider the following joint feature representation for a couple: Φ(ei) = Φ(xi, x′

i) = Ψ(xi, x′ i) − Ψ(x′ i, xi),

This yields the following kernel defined on couples: K Φ(ei, ej) = K Φ(xi, x′

i, xj, x′ j)

= Ψ(xi, x′

i) − Ψ(x′ i, xi), Ψ(xj, x′ j) − Ψ(x′ j, xj)

= K Ψ(xi, x′

i, xj, x′ j) + K Ψ(x′ i, xi, x′ j, xj)

−K Ψ(x′

i, xi, xj, x′ j) − K Ψ(xi, x′ i, x′ j, xj) .

And the model becomes: h(x, x′) = w, Ψ(x, x′) − Ψ(x′, x) =

N

• i=1

aiK Φ(xi, x′

i, x, x′) .

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 11 / 25

Learning intransitive reciprocal relations

Ranking can be considered as a specific case in this framework

Consider the following joint feature representation Ψ for a couple: ΨT(x, x′) = φ(x) . This yields the following kernel K Ψ: K Ψ

T (xi, x′ i, xj, x′ j) = K φ(xi, xj) = φ(xi), φ(xj) ,

And the model becomes: h(x, x′) = w, φ(x) − w, φ(x′) = f(x) − f(x′) ,

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 12 / 25

Learning intransitive reciprocal relations

Ranking can be considered as a specific case in this framework

Consider the following joint feature representation Ψ for a couple: ΨT(x, x′) = φ(x) . This yields the following kernel K Ψ: K Ψ

T (xi, x′ i, xj, x′ j) = K φ(xi, xj) = φ(xi), φ(xj) ,

And the model becomes: h(x, x′) = w, φ(x) − w, φ(x′) = f(x) − f(x′) ,

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 12 / 25

Learning intransitive reciprocal relations

Ranking can be considered as a specific case in this framework

Consider the following joint feature representation Ψ for a couple: ΨT(x, x′) = φ(x) . This yields the following kernel K Ψ: K Ψ

T (xi, x′ i, xj, x′ j) = K φ(xi, xj) = φ(xi), φ(xj) ,

And the model becomes: h(x, x′) = w, φ(x) − w, φ(x′) = f(x) − f(x′) ,

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 12 / 25

Learning intransitive reciprocal relations

Using the Kronecker-product intransitive relations can be learned, unlike the existing approaches

Consider the following joint feature representation Ψ for a couple: ΨI(x, x′) = φ(x) ⊗ φ(x′) , where ⊗ denotes the Kronecker-product: A ⊗ B =    A1,1B · · · A1,nB . . . ... . . . Am,1B · · · Am,nB    , This yields the following kernel K Ψ: K Ψ

I (xi, x′ i, xj, x′ j)

= φ(xi) ⊗ φ(x′

i), φ(xj) ⊗ φ(x′ j)

= φ(xi), φ(xj) ⊗ φ(x′

i), φ(x′ j)

= K φ(xi, xj)K φ(x′

i, x′ j),

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 13 / 25

Learning intransitive reciprocal relations

Using the Kronecker-product intransitive relations can be learned, unlike the existing approaches

Consider the following joint feature representation Ψ for a couple: ΨI(x, x′) = φ(x) ⊗ φ(x′) , where ⊗ denotes the Kronecker-product: A ⊗ B =    A1,1B · · · A1,nB . . . ... . . . Am,1B · · · Am,nB    , This yields the following kernel K Ψ: K Ψ

I (xi, x′ i, xj, x′ j)

= φ(xi) ⊗ φ(x′

i), φ(xj) ⊗ φ(x′ j)

= φ(xi), φ(xj) ⊗ φ(x′

i), φ(x′ j)

= K φ(xi, xj)K φ(x′

i, x′ j),

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 13 / 25

Experiments

Outline

1

Introduction

2

Stochastic transitivity and ranking representability

3

Learning intransitive reciprocal relations

4

Experiments

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 14 / 25

Experiments

Reciprocal relations in rock-paper-scissors

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 15 / 25

Experiments

Reciprocal relations in rock-paper-scissors

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 16 / 25

Experiments

Reciprocal relations in rock-paper-scissors

Convert probabilities to a reciprocal relation: Q(x, x′) = P(r | x)iP(s | x′) + 1 2P(r | x)P(r | x′) +P(p | x)P(r | x′) + 1 2P(p | x)P(p | x′) +P(s | x)P(p | x′) + 1 2P(s | x)P(s | x′). Example: Player1 :x = (r = 1/2, p = 1/2, s = 0) Player2 :x′ = (r = 0, p = 1/2, s = 1/2) ⇒ Q(x, x′) = 1/2(1/2 + 0/2) + 1/2(0 + 1/4) + 0(1/2 + 1/4) = 3/8

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 17 / 25

Experiments

Rock-paper-scissors: experimental setup

100 players for training (100 games) 100 players for testing (1000 games) features are the mixed strategies training labels y ∈ {−1, 0, 1} test labels y ∈ [0, 1] K φ linear kernel three different settings

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 18 / 25

Experiments

Rock-paper-scissors: experimental setup

100 players for training (100 games) 100 players for testing (1000 games) features are the mixed strategies training labels y ∈ {−1, 0, 1} test labels y ∈ [0, 1] K φ linear kernel three different settings

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 18 / 25

Experiments

Rock-paper-scissors: experimental setup

100 players for training (100 games) 100 players for testing (1000 games) features are the mixed strategies training labels y ∈ {−1, 0, 1} test labels y ∈ [0, 1] K φ linear kernel three different settings

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 18 / 25

Experiments

Rock-paper-scissors: experimental setup

100 players for training (100 games) 100 players for testing (1000 games) features are the mixed strategies training labels y ∈ {−1, 0, 1} test labels y ∈ [0, 1] K φ linear kernel three different settings

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 18 / 25

Experiments

Rock-paper-scissors: experimental setup

100 players for training (100 games) 100 players for testing (1000 games) features are the mixed strategies training labels y ∈ {−1, 0, 1} test labels y ∈ [0, 1] K φ linear kernel three different settings

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 18 / 25

Experiments

Rock-paper-scissors: setting 1

Rock Paper Scissors

Method MSE Intrans. 0.000209 Trans. 0.000162 Naive 0.000001

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 19 / 25

Experiments

Rock-paper-scissors: setting 1

Rock Paper Scissors

Method MSE Intrans. 0.000209 Trans. 0.000162 Naive 0.000001

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 19 / 25

Experiments

Rock-paper-scissors: setting 2

Rock Paper Scissors

Method MSE Intrans. 0.000445 Trans. 0.006804 Naive 0.006454

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 20 / 25

Experiments

Rock-paper-scissors: setting 2

Rock Paper Scissors

Method MSE Intrans. 0.000445 Trans. 0.006804 Naive 0.006454

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 20 / 25

Experiments

Rock-paper-scissors: setting 3

Rock Paper Scissors

Method MSE Intrans. 0.000076 Trans. 0.131972 Naive 0.125460

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 21 / 25

Experiments

Rock-paper-scissors: setting 3

Rock Paper Scissors

Method MSE Intrans. 0.000076 Trans. 0.131972 Naive 0.125460

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 21 / 25

Experiments

Simulation of competition between species results in stable populations after many iterations

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 22 / 25

Experiments

Experiment 2: competition between species in theoretical biology

0.0 0.2 0.4 0.6 0.8 1.0 Strong point 0.0 0.2 0.4 0.6 0.8 1.0 Weak point 0.0 0.2 0.4 0.6 0.8 1.0 Strong point 0.0 0.2 0.4 0.6 0.8 1.0 Weak point

y = sign(d(s(x′), w(x)) − d(s(x), w(x′)))

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 23 / 25

Experiments

The intransitive kernel clearly beats the traditional transitive kernel

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

• Trans. Accuracy = 0.615 ⇔ Intrans. Accuracy = 0.850

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 24 / 25

Experiments

Discussion

Existing kernel-based ranking methods cannot predict intransitive relations. With our framework it is possible to represent and predict intransitive relations in an adequate way. Empirical results on two problems confirm that our framework is able to learn intransitive relations, unlike ranking methods. Many applications possible (e.g. in the life sciences), but no publicly available datasets. http://staff.cs.utu.fi/˜ aatapa/software/RPS

Pahikkala et al. (TUCS, UGent) Intransitive Preference Learning September, 11 2009 25 / 25