Interval-valued regression and classication models in the framework - - PowerPoint PPT Presentation

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

Interval-valued regression and classi…cation models in the framework of machine learning

Lev Utkin and Frank Coolen Innsbruck, July 2011

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

A general problem statement

Given: a training set (xi, yi), i = 1, ..., n, x 2 Rm is a multivariate input of features and a scalar output:

regression: y 2 R classi…cation: binary y 2 f1, 1g or multi-class y 2 f1, 2, ..., lg.

The learning problem: to select a function f (x, wopt) from a set of functions f (x, w) parameterized by a set of parameters w 2 Λ, which

regression: best approximates the system response y classi…cation: separates examples of di¤erent classes y.

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

A general problem solution

To minimize the risk functional R(w) over w 2 Λ: regression R(w) =

Z

Rm+1 L(y, f )dF0(x, y)

=

Z

R L(z, w)dF(z), z = y f (x, w).

classi…cation R(w) =

Z

Rmf1,1g L(y, f )dF0(x, y)

= ∑

y=0,1

p(y)

Z

Rm L(y, f )dF0(x j y).

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

Loss functions

in regression models: quadratic, linear, the “pinball” function (for quantile regression), the ε-insensitive loss function. in classi…cation models: indicator, logistic, hinge loss.

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

The main idea

1

It is assumed that the CDF F(z) 2 F bounded by the lower F and upper F CDFs (P-boxes): F = fF(z) j 8z, F(z j y) F(z j y) F(z j y)g.

2

P-boxes are constructed from training data and they are parametric, i.e., F ! F(w).

3

Two CDFs maximizing and minimizing R(w) are taken from F, which determine the largest R and smallest R risk measures as functions of w.

4

w are computed by minimizing the lower and upper risk measures.

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

Two main tasks to be solved

1

How to construct parametric P-boxes, i.e., F and F from the training set?

2

How to …nd “optimal” distributions from the P-box, i.e., the distributions maximizing and minimizing the risk functional (corresponding to minimax and minimin strategies, respectively)?

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

Interval regression (simplest case) x1 y f

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

Intervals and P-boxes (regression)

Given: a training set (xi, Yi), i = 1, ..., n, x 2 Rm, Yi = [yi, yi]. P-boxes: F(z j w) = Bel((∞, z]) = n1

∑

i:Zi (w )z

1, F(z j w) = Pl((∞, z]) = n1

∑

i:Zi (w )z

1. Here Zi(w) = yi f (xi, w) and Zi(w) = yi f (xi, w).

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

“Optimal distribution functions” (regression) (Utkin & Destercke 2009)

z

1 2 3 4

Interval-valued estimates Zi(w), Zi(w)

• Lev Utkin and Frank Coolen

Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

Lower and upper CDFs

• 8
• 6
• 4
• 2

2 4 6 8 0.2 0.4 0.6 0.8 1.0

z F

lower F upper F

Lower and upper probability distributions produced by four intervals

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

The optimal CDF by the minimax strategy

• 8
• 6
• 4
• 2

2 4 6 8 0.2 0.4 0.6 0.8 1.0

z F

lower F upper F

The optimal probability distribution (thick) by the minimax strategy

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

The optimal CDF by the minimin strategy

• 8
• 6
• 4
• 2

2 4 6 8 0.2 0.4 0.6 0.8 1.0

z F

lower F upper F

The optimal probability distribution (thick) by the minimin strategy

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

Intervals and expectations in the framework of belief structures (regression)

The upper expectation of risk functional (Nguyen & Walker 1994, Strat 1990): R(w) = n1 ∑

n i=1

max

z2[Zi (w ),Zi (w )]

L(z) ! min

w .

The optimization problem for computing w: min

w ,Gi ∑ n i=1 Gi,

subject to Gi L(Zi(w)), Gi L(Zi(w)), i = 1, ..., n. If L(z) and f (x, w) are linear, then we have the LP problem.

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

Support vector machine (SVM) and interval observations

If we take the ε-insensitive loss function L, then min

α

1 2 hα, αi + C ∑

n i=1 (ξi + ξ i )

• subject to

ξi 0, ξ

i 0,

ξi + ε (hαxii + α0) yi, ξi + ε (hαxii + α0) yi, ξ

i + ε yi (hαxii + α0) , ξ i + ε yi (hαxii + α0) . 1 2 hα, αi is the Tikhonov regularization term (the most popular

penalty or smoothness term)

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

Advantages of SVMs

1

SVMs are ‡exible in the choice of the form of the discriminant and regression functions (non-linear functions f due to kernel methodology);

2

SVMs provide a unique solution (due to convex objective function), there are no false local minima;

3

SVMs are simple to use;

4

SVMs have a clear geometric explanation.

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

Interval data in classi…cation x1 x2 f

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

What do the minimax and minimin strategies mean?

1

Regression:

minimax: outlying points are taken into account; minimin: neighboring points are taken into account.

2

Classi…cation:

minimax: points from two classes approach each other (get mixed); minimin: points are separated.

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew

A statement of the standard machine learning problem A new approach for regression and classi…cation modeling Regression models Classi…cation models

Questions

?

Lev Utkin and Frank Coolen Interval-valued regression and classi…cation models in the framew