First International Workshop on Learning over Multiple Contexts LMCE - - PowerPoint PPT Presentation

• C. Ferri, J. Hernández-Orallo, A. Martínez-Usó and M.J. Ramírez-Quintana

DSIC, UPV, DSIC, UPV, València València, Spain , Spain {cferri, jorallo, admarus, mramirez}@dsic.upv.es

First International Workshop on Learning over Multiple Contexts LMCE 2014

Nancy, 19 September 2014

First International Workshop on Learning over Multiple Contexts, LMCE 2014

 Motivation  The Noise setting  Context Plots and Dominance  Experiments  Conclusions and Future Work

2

First International Workshop on Learning over Multiple Contexts, LMCE 2014 3

• Very often, operating contexts (OC) at the

training and the deployment time are different.

• The error of the model depends on the level of

noise introduced by the OC.

training deployment Operating Context Q features idealistic “perfect” “noisy” less-idealistic

First International Workshop on Learning over Multiple Contexts, LMCE 2014 4

• Alarm system where two models (A and B) have been

trained (ideal conds., OC=temperature in [0,30])

• Validation: A>B (A is better!)
• Deployment
• How these OC affect to sensors is also known
• OC in deployment are given
• Which model is better for each OC? …. For several

OC, model B could be better now!

First International Workshop on Learning over Multiple Contexts, LMCE 2014

In order to answer this question we propose:

 To evaluate the models with different levels of

simulate noise.

 To draw a context plot with all models, and to

determine dominance regions.

 During deployment, the noise level is derived from

the OC and the best model for that noise is applied.

#

First International Workshop on Learning over Multiple Contexts, LMCE 2014

Noise is calculated by using probability distributions:

 Numerical attributes

• we estimate the σ of all values of the attribute
• for a level of noise ν, we modify a value x using a normal

distribution x’ ~ N(x, σ . ν)

 Nominal attribute

• we estimate the frecuency of each value vi, p=(pv1,…,pvn)
• for an instance x with value vi we estimate the vector

t=(t1,…,tn) ti=1 and tj=0 if i≠j

• for a level of noise ν we calculate p’= α . p + (1-α) . t where

α=1-e(-ν)

• we use p’ to sample the new value x’

#

First International Workshop on Learning over Multiple Contexts, LMCE 2014 7

Vehicle Vehicle

First International Workshop on Learning over Multiple Contexts, LMCE 2014 8

• 12 datasets from the UCI repository

12 datasets from the UCI repository

• 50% Train, 25% Validation, 25 % Test

50% Train, 25% Validation, 25 % Test

• Classification:

Classification:

• J48, Naive Bayes, Logistic Regression and

J48, Naive Bayes, Logistic Regression and kNN kNN. .

• Reference method: majority class.

Reference method: majority class.

• Classification error

Classification error

• Regression:

Regression:

• Linear Regression,M5P,

Linear Regression,M5P, kNN kNN , , SMOreg SMOreg. .

• Reference method:

Reference method: ZeroR ZeroR. .

• Relative absolute error

Relative absolute error

Methodology: Methodology:

First International Workshop on Learning over Multiple Contexts, LMCE 2014 9

Creditg Creditg

First International Workshop on Learning over Multiple Contexts, LMCE 2014 10

Abalone Abalone

First International Workshop on Learning over Multiple Contexts, LMCE 2014 11

• ValNoNoise

ValNoNoise: For all the estimated values of noise, we select the method that obtains the best performance without noise.

• ValBestArea

ValBestArea: For all the estimated values of noise, we select the method that obtains the best performance in the validation dataset by averaging all noise levels (i.e., the curve with lowest area in the context plot).

• ValNoiseOpt

ValNoiseOpt: For each of the estimated values of noise, we select the method with best performance in the validation dataset with that value of noise.

• Idealistic

Idealistic: For each of the estimated values of noise, we select the model with best performance in the test dataset with that value of noise. This strategy is not realistic (it cannot be done in practice). We just include this as a reference

Methods: Methods:

First International Workshop on Learning over Multiple Contexts, LMCE 2014 12

• The performance results are

The performance results are normalised normalised by the Idealistic by the Idealistic performance performance (idealistic/method)*100 (idealistic/method)*100

First International Workshop on Learning over Multiple Contexts, LMCE 2014 13

1.

• 1. In this paper we have analysed the case when:

In this paper we have analysed the case when:

 the noise level depends on a context

the noise level depends on a context

 we know the context in advance

we know the context in advance

2.

• 2. The model that best behaves for each noise level situation is used.

The model that best behaves for each noise level situation is used.

3.

• 3. CONS:

CONS: It It takes takes some some time.

• time. PROS:

PROS: Selection/application Selection/application of

• f the

the best best model is straightforward with results close to an idealistic process. model is straightforward with results close to an idealistic process.

4.

• 4. As

As a a future future work work we we plan plan working working on

• n different

different noise noise models, models, derived derived from scenarios with real operating conditions and with real sensors. from scenarios with real operating conditions and with real sensors.

• Each

Each attribute attribute will will have have a a different different operating

• perating range,

range, but but the the context would still be given by a single parameter (e.g., , temperature, or the number of measurements performed..). temperature, or the number of measurements performed..).

First International Workshop on Learning over Multiple Contexts, LMCE 2014

Thanks for your atention…..

#