Suboptimality of Penalized Empirical Risk Minimization in - - PowerPoint PPT Presentation

▶
suboptimality of penalized empirical risk minimization in
SMART_READER_LITE
LIVE PREVIEW

Suboptimality of Penalized Empirical Risk Minimization in - - PowerPoint PPT Presentation

Jul 30, 2023 161 likes •711 views

Suboptimality of Penalized Empirical Risk Minimization in Classification. Guillaume Lecu e Universit e Paris 6 COLT 2007, June 13 General Framework. Aggregations Procedures. Optimality in classification. Motivation. M prior estimators

slide-1
SLIDE 1

Suboptimality of Penalized Empirical Risk Minimization in Classification.

Guillaume Lecu´ e

Universit´ e Paris 6

COLT 2007, June 13

slide-2
SLIDE 2

General Framework. Aggregations Procedures. Optimality in classification.

Motivation. M prior estimators (’weak’ estimators) : f1, . . . , fM n observations : Dn

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-3
SLIDE 3

General Framework. Aggregations Procedures. Optimality in classification.

Motivation. M prior estimators (’weak’ estimators) : f1, . . . , fM n observations : Dn

Aim

Construction of a new estimator which is approximatively as good as the best ’weak’ estimator : Aggregation method or Aggregate

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-4
SLIDE 4

General Framework. Aggregations Procedures. Optimality in classification.

Examples. Adaptation : Observations : Dm+n Estimation : Dm →non-adaptive estimators f1, . . . , fM. learning : D(n) →aggregate ˜ fn (adaptive).

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-5
SLIDE 5

General Framework. Aggregations Procedures. Optimality in classification.

Examples. Adaptation : Observations : Dm+n Estimation : Dm →non-adaptive estimators f1, . . . , fM. learning : D(n) →aggregate ˜ fn (adaptive). Estimation : ǫ−net : f1, . . . , fM (functions) learning : Dn →aggregate ˜ fn.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-6
SLIDE 6

General Framework. Aggregations Procedures. Optimality in classification.

Model of classification (X, A) a measurable space, (X, Y ) ∼ π valued in X × {−1, 1},

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-7
SLIDE 7

General Framework. Aggregations Procedures. Optimality in classification.

Model of classification (X, A) a measurable space, (X, Y ) ∼ π valued in X × {−1, 1}, Dn = ((X1, Y1), . . . , (Xn, Yn)) : n i.i.d. observations.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-8
SLIDE 8

General Framework. Aggregations Procedures. Optimality in classification.

Model of classification (X, A) a measurable space, (X, Y ) ∼ π valued in X × {−1, 1}, Dn = ((X1, Y1), . . . , (Xn, Yn)) : n i.i.d. observations. Problem of prediction : x ∈ X → label y ∈ {−1, 1} ?

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-9
SLIDE 9

General Framework. Aggregations Procedures. Optimality in classification.

Model of classification (X, A) a measurable space, (X, Y ) ∼ π valued in X × {−1, 1}, Dn = ((X1, Y1), . . . , (Xn, Yn)) : n i.i.d. observations. Problem of prediction : x ∈ X → label y ∈ {−1, 1} ? f : X − → {−1, 1} : prediction rule.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-10
SLIDE 10

General Framework. Aggregations Procedures. Optimality in classification.

Model of classification (X, A) a measurable space, (X, Y ) ∼ π valued in X × {−1, 1}, Dn = ((X1, Y1), . . . , (Xn, Yn)) : n i.i.d. observations. Problem of prediction : x ∈ X → label y ∈ {−1, 1} ? f : X − → {−1, 1} : prediction rule. Bayes risk : A0(f ) = P[f (X) = Y ]

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-11
SLIDE 11

General Framework. Aggregations Procedures. Optimality in classification.

Model of classification (X, A) a measurable space, (X, Y ) ∼ π valued in X × {−1, 1}, Dn = ((X1, Y1), . . . , (Xn, Yn)) : n i.i.d. observations. Problem of prediction : x ∈ X → label y ∈ {−1, 1} ? f : X − → {−1, 1} : prediction rule. Bayes risk : A0(f ) = P[f (X) = Y ] Bayes rule : f ∗(x) = Sign(2η(x) − 1) where η(x) = P[Y = 1|X = x]. A∗

def

= minf A0(f ) = A0(f ∗)

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-12
SLIDE 12

General Framework. Aggregations Procedures. Optimality in classification.

Model of classification (X, A) a measurable space, (X, Y ) ∼ π valued in X × {−1, 1}, Dn = ((X1, Y1), . . . , (Xn, Yn)) : n i.i.d. observations. Problem of prediction : x ∈ X → label y ∈ {−1, 1} ? f : X − → {−1, 1} : prediction rule. Bayes risk : A0(f ) = P[f (X) = Y ] Bayes rule : f ∗(x) = Sign(2η(x) − 1) where η(x) = P[Y = 1|X = x]. A∗

def

= minf A0(f ) = A0(f ∗) Prediction→estimation : estimation of f ∗.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-13
SLIDE 13

General Framework. Aggregations Procedures. Optimality in classification.

Model of classification (X, A) a measurable space, (X, Y ) ∼ π valued in X × {−1, 1}, Dn = ((X1, Y1), . . . , (Xn, Yn)) : n i.i.d. observations. Problem of prediction : x ∈ X → label y ∈ {−1, 1} ? f : X − → {−1, 1} : prediction rule. Bayes risk : A0(f ) = P[f (X) = Y ] Bayes rule : f ∗(x) = Sign(2η(x) − 1) where η(x) = P[Y = 1|X = x]. A∗

def

= minf A0(f ) = A0(f ∗) Prediction→estimation : estimation of f ∗. excess risk : A0(f ) − A∗

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-14
SLIDE 14

General Framework. Aggregations Procedures. Optimality in classification.

Model of classification (f : X − → R) → risk A0(f ) = E[φ0(Yf (X))] where φ0(x) = 1 I(x≤0) classical loss or 0 − 1 loss

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-15
SLIDE 15

General Framework. Aggregations Procedures. Optimality in classification.

Model of classification (f : X − → R) → risk A0(f ) = E[φ0(Yf (X))] where φ0(x) = 1 I(x≤0) classical loss or 0 − 1 loss φ1(x) = max(0, 1 − x) hinge loss or (SVM loss) x − → log2(1 + exp(−x)) ’Logit-Boosting’ loss x − → exp(−x) exponential Boosting loss x − → (1 − x)2 quadratic loss x − → max(0, 1 − x)2 2-norm soft margin loss

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-16
SLIDE 16

General Framework. Aggregations Procedures. Optimality in classification.

Model of classification (f : X − → R) → risk A0(f ) = E[φ0(Yf (X))] where φ0(x) = 1 I(x≤0) classical loss or 0 − 1 loss φ1(x) = max(0, 1 − x) hinge loss or (SVM loss) x − → log2(1 + exp(−x)) ’Logit-Boosting’ loss x − → exp(−x) exponential Boosting loss x − → (1 − x)2 quadratic loss x − → max(0, 1 − x)2 2-norm soft margin loss φ−risk : Aφ(f ) = E[φ(Yf (X))], Aφ∗ def = inff A(f ) = A(f φ∗), excess φ−risk : Aφ(f ) − Aφ∗. empirical φ−risk : Aφ

n(f ) = 1

n

n

  • i=1

φ(Yif (Xi)).

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-17
SLIDE 17

General Framework. Aggregations Procedures. Optimality in classification.

Selectors. φ : R − → R a loss, F0 = {f1, . . . , fM} ⊂ F a dictionary. Empirical Risk Minimization (ERM) :(Vapnik, Chervonenkis...) ˜ f ERM

n

∈ Arg min

f ∈F0 Aφ n(f ).

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-18
SLIDE 18

General Framework. Aggregations Procedures. Optimality in classification.

Selectors. φ : R − → R a loss, F0 = {f1, . . . , fM} ⊂ F a dictionary. Empirical Risk Minimization (ERM) :(Vapnik, Chervonenkis...) ˜ f ERM

n

∈ Arg min

f ∈F0 Aφ n(f ).

penalized Empirical Risk Minimization (pERM) : ˜ f ERM

n

∈ Arg min

f ∈F0[Aφ n(f ) + pen(f )],

where pen is a penalty function. (Barron, Bartlett, Birg´ e, Boucheron, Koltchinski, Lugosi, Massart,...)

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-19
SLIDE 19

General Framework. Aggregations Procedures. Optimality in classification.

Aggregation methods with exponential weights. φ : R − → R a loss, F0 = {f1, . . . , fM} ⊂ F a dictionary. Aggregate with Exponential weights (AEW) : ˜ f AEW

n,T

=

  • f ∈F0

w (n)

T (f )f , where w (n) T (f ) =

exp

  • −nTAφ

n(f )

  • g∈F0 exp
  • −nTAφ

n(g)

, T −1 : temperature parameter.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-20
SLIDE 20

General Framework. Aggregations Procedures. Optimality in classification.

Aggregation methods with exponential weights. φ : R − → R a loss, F0 = {f1, . . . , fM} ⊂ F a dictionary. Aggregate with Exponential weights (AEW) : ˜ f AEW

n,T

=

  • f ∈F0

w (n)

T (f )f , where w (n) T (f ) =

exp

  • −nTAφ

n(f )

  • g∈F0 exp
  • −nTAφ

n(g)

, T −1 : temperature parameter. Cumulative Aggregate with Exponential Weights (CAEW) :(Catoni, Yang,...) ˜ f CAEW

n,T

= 1 n

n

  • k=1

˜ f AEW

k,T

.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-21
SLIDE 21

General Framework. Aggregations Procedures. Optimality in classification.

Aim of Aggregation(1) : Optimal rate of aggregation.

Definition

∀F0 = {f1, . . . , fM} ⊆ F, ∃˜ fn such that ∀π ∈ P, ∀n ≥ 1 E

  • A(˜

fn) − A∗ ≤ min

f ∈F0 (A(f ) − A∗) + C0γ(n, M).

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-22
SLIDE 22

General Framework. Aggregations Procedures. Optimality in classification.

Aim of Aggregation(1) : Optimal rate of aggregation.

Definition

∀F0 = {f1, . . . , fM} ⊆ F, ∃˜ fn such that ∀π ∈ P, ∀n ≥ 1 E

  • A(˜

fn) − A∗ ≤ min

f ∈F0 (A(f ) − A∗) + C0γ(n, M).

∃F0 = {f1, . . . , fM} such that for any aggregate ¯ fn, ∃π ∈ P, ∀n ≥ 1 E

  • A(¯

fn) − A∗ ≥ min

f ∈F0 (A(f ) − A∗) + C1γ(n, M).

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-23
SLIDE 23

General Framework. Aggregations Procedures. Optimality in classification.

Aim of Aggregation(1) : Optimal rate of aggregation.

Definition

∀F0 = {f1, . . . , fM} ⊆ F, ∃˜ fn such that ∀π ∈ P, ∀n ≥ 1 E

  • A(˜

fn) − A∗ ≤ min

f ∈F0 (A(f ) − A∗) + C0γ(n, M).

∃F0 = {f1, . . . , fM} such that for any aggregate ¯ fn, ∃π ∈ P, ∀n ≥ 1 E

  • A(¯

fn) − A∗ ≥ min

f ∈F0 (A(f ) − A∗) + C1γ(n, M).

γ(n, M) is an optimal rate of aggregation and ˜ fn is an optimal aggregation procedure.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-24
SLIDE 24

General Framework. Aggregations Procedures. Optimality in classification.

Aim of Aggregation(2) : Adaptation.

Definition (Oracle Inequality)

∀F0 = {f1, . . . , fM} ⊆ F, ∃˜ fn such that ∀π ∈ P, ∀n ≥ 1 E

  • A(˜

fn) − A∗ ≤ C min

f ∈F0 (A(f ) − A∗) + C0γ(n, M),

where C ≥ 1.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-25
SLIDE 25

General Framework. Aggregations Procedures. Optimality in classification.

Continuous scale of loss functions. Classification problem : Aφ(f ) = E[φ(Yf (X))], Y ∈ {−1, 1}, X ∈ X. φ(x) = φh(x) = (1 − h)φ0(x) + hφ1(x) if 0 ≤ h ≤ 1 (h − 1)x2 − x + 1 if h > 1, ∀x ∈ R where φ0(z) = 1 I(z≤0) is the 0 − 1 loss and φ1(z) = max(0, 1 − z) is the hinge loss. h = 0

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-26
SLIDE 26

General Framework. Aggregations Procedures. Optimality in classification.

Continuous scale of loss functions. Classification problem : Aφ(f ) = E[φ(Yf (X))], Y ∈ {−1, 1}, X ∈ X. φ(x) = φh(x) =

  • (1 − h)φ0(x) + hφ1(x)

if 0 ≤ h ≤ 1 (h − 1)x2 − x + 1 if h > 1, ∀x ∈ R where φ0(z) = 1 I(z≤0) is the 0 − 1 loss and φ1(z) = max(0, 1 − z) is the hinge loss. h = 1/3

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-27
SLIDE 27

General Framework. Aggregations Procedures. Optimality in classification.

Continuous scale of loss functions. Classification problem : Aφ(f ) = E[φ(Yf (X))], Y ∈ {−1, 1}, X ∈ X. φ(x) = φh(x) =

  • (1 − h)φ0(x) + hφ1(x)

if 0 ≤ h ≤ 1 (h − 1)x2 − x + 1 if h > 1, ∀x ∈ R where φ0(z) = 1 I(z≤0) is the 0 − 1 loss and φ1(z) = max(0, 1 − z) is the hinge loss. h = 2/3

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-28
SLIDE 28

General Framework. Aggregations Procedures. Optimality in classification.

Continuous scale of loss functions. Classification problem : Aφ(f ) = E[φ(Yf (X))], Y ∈ {−1, 1}, X ∈ X. φ(x) = φh(x) = (1 − h)φ0(x) + hφ1(x) if 0 ≤ h ≤ 1 (h − 1)x2 − x + 1 if h > 1, ∀x ∈ R where φ0(z) = 1 I(z≤0) is the 0 − 1 loss and φ1(z) = max(0, 1 − z) is the hinge loss. h = 1

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-29
SLIDE 29

General Framework. Aggregations Procedures. Optimality in classification.

Continuous scale of loss functions. Classification problem : Aφ(f ) = E[φ(Yf (X))], Y ∈ {−1, 1}, X ∈ X. φ(x) = φh(x) = (1 − h)φ0(x) + hφ1(x) if 0 ≤ h ≤ 1 (h − 1)x2 − x + 1 if h > 1, ∀x ∈ R where φ0(z) = 1 I(z≤0) is the 0 − 1 loss and φ1(z) = max(0, 1 − z) is the hinge loss. h = 2

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-30
SLIDE 30

General Framework. Aggregations Procedures. Optimality in classification.

Continuous scale of loss functions. Classification problem : Aφ(f ) = E[φ(Yf (X))], Y ∈ {−1, 1}, X ∈ X. φ(x) = φh(x) = (1 − h)φ0(x) + hφ1(x) if 0 ≤ h ≤ 1 (h − 1)x2 − x + 1 if h > 1, ∀x ∈ R where φ0(z) = 1 I(z≤0) is the 0 − 1 loss and φ1(z) = max(0, 1 − z) is the hinge loss. h = 3

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-31
SLIDE 31

General Framework. Aggregations Procedures. Optimality in classification.

ORA in classification Loss function 0 ≤ h < 1 h = 1 h > 1 Optimal rate of aggregation (ORA)

  • log M

n

  • log M

n log M n

Optimal aggregation proce- dure ERM ERM, AEW, CAEW CAEW

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-32
SLIDE 32

General Framework. Aggregations Procedures. Optimality in classification.

ORA in classification Loss function 0 ≤ h < 1 h = 1 h > 1 Optimal rate of aggregation (ORA)

  • log M

n

  • log M

n log M n

Optimal aggregation proce- dure ERM ERM, AEW, CAEW CAEW

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-33
SLIDE 33

General Framework. Aggregations Procedures. Optimality in classification.

ORA in classification Loss function 0 ≤ h < 1 h = 1 h > 1 Optimal rate of aggregation (ORA)

  • log M

n

  • log M

n log M n

Optimal aggregation proce- dure ERM ERM, AEW, CAEW CAEW

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-34
SLIDE 34

General Framework. Aggregations Procedures. Optimality in classification.

ORA in classification Loss function 0 ≤ h < 1 h = 1 h > 1 Optimal rate of aggregation (ORA)

  • log M

n

  • log M

n log M n

Optimal aggregation proce- dure ERM ERM, AEW, CAEW CAEW

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-35
SLIDE 35

General Framework. Aggregations Procedures. Optimality in classification.

ORA in classification Loss function 0 ≤ h < 1 h = 1 h > 1 Optimal rate of aggregation (ORA)

  • log M

n

  • log M

n log M n

Optimal aggregation proce- dure ERM ERM, AEW, CAEW CAEW

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-36
SLIDE 36

General Framework. Aggregations Procedures. Optimality in classification.

ORA in classification Loss function 0 ≤ h < 1 h = 1 h > 1 Optimal rate of aggregation (ORA)

  • log M

n

  • log M

n log M n

Optimal aggregation proce- dure ERM ERM, AEW, CAEW CAEW

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-37
SLIDE 37

General Framework. Aggregations Procedures. Optimality in classification.

2 Questions. Question 1 : Why is there such a breakdown just after the Hinge loss ?

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-38
SLIDE 38

General Framework. Aggregations Procedures. Optimality in classification.

2 Questions. Question 1 : Why is there such a breakdown just after the Hinge loss ? 0 ≤ h ≤ 1,

  • log M

n − → log M n , h > 1.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-39
SLIDE 39

General Framework. Aggregations Procedures. Optimality in classification.

2 Questions. Question 1 : Why is there such a breakdown just after the Hinge loss ? 0 ≤ h ≤ 1,

  • log M

n − → log M n , h > 1. Question 2 : Do we really need aggregation procedures with exponential weights to achieve the optimal rates of aggregation ?

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-40
SLIDE 40

General Framework. Aggregations Procedures. Optimality in classification.

2 Questions. Question 1 : Why is there such a breakdown just after the Hinge loss ? 0 ≤ h ≤ 1,

  • log M

n − → log M n , h > 1. ERM − → CAEW Question 2 : Do we really need aggregation procedures with exponential weights to achieve the optimal rates of aggregation ?

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-41
SLIDE 41

General Framework. Aggregations Procedures. Optimality in classification.

Question 1. Why there is a breakdown at h = 1 ?

Margin assumption for the loss function φ :

The probability measure π satisfies the φ−margin assumption φ−MA(κ), with margin parameter κ ≥ 1 if E[(φ(Yf (X)) − φ(Yf φ∗(X)))2] ≤ cφ(Aφ(f ) − Aφ∗)1/κ, for any f : X − → R.

  • cf. Mammen and Tsybakov 99 (discriminant analysis) and Tsybakov 04

(classification).

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-42
SLIDE 42

General Framework. Aggregations Procedures. Optimality in classification.

Question 1. Why there is a breakdown at h = 1 ?

Margin assumption for the loss function φ :

The probability measure π satisfies the φ−margin assumption φ−MA(κ), with margin parameter κ ≥ 1 if E[(φ(Yf (X)) − φ(Yf φ∗(X)))2] ≤ cφ(Aφ(f ) − Aφ∗)1/κ, for any f : X − → R.

  • cf. Mammen and Tsybakov 99 (discriminant analysis) and Tsybakov 04

(classification). φ0 − MA(κ) ⇐ ⇒ P[|2η(X) − 1| ≤ t] ≤ tα, ∀0 < t < 1, α = 1 κ − 1 η(x) = P[Y = 1|X = x]

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-43
SLIDE 43

General Framework. Aggregations Procedures. Optimality in classification.

Question 1. Why there is a breakdown at h = 1 ?

Margin assumption for the loss function φ :

The probability measure π satisfies the φ−margin assumption φ−MA(κ), with margin parameter κ ≥ 1 if E[(φ(Yf (X)) − φ(Yf φ∗(X)))2] ≤ cφ(Aφ(f ) − Aφ∗)1/κ, for any f : X − → R.

  • cf. Mammen and Tsybakov 99 (discriminant analysis) and Tsybakov 04

(classification). φ0 − MA(κ) ⇐ ⇒ P[|2η(X) − 1| ≤ t] ≤ tα, ∀0 < t < 1, α = 1 κ − 1 η(x) = P[Y = 1|X = x] (κ = 1 ⇐ ⇒ ∃h > 0, |2η(X) − 1| ≥ h)

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-44
SLIDE 44

General Framework. Aggregations Procedures. Optimality in classification.

Question 1. Why there is a breakdown at h = 1 ? κ = +∞ for any 0 ≤ h ≤ 1.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-45
SLIDE 45

General Framework. Aggregations Procedures. Optimality in classification.

Question 1. Why there is a breakdown at h = 1 ? κ = +∞ for any 0 ≤ h ≤ 1.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-46
SLIDE 46

General Framework. Aggregations Procedures. Optimality in classification.

Question 1. Why there is a breakdown at h = 1 ? κ = +∞ for any 0 ≤ h ≤ 1.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-47
SLIDE 47

General Framework. Aggregations Procedures. Optimality in classification.

Question 1. Why there is a breakdown at h = 1 ? κ = +∞ for any 0 ≤ h ≤ 1.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-48
SLIDE 48

General Framework. Aggregations Procedures. Optimality in classification.

Question 1. Why there is a breakdown at h = 1 ? κ = 1 for any h > 1.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-49
SLIDE 49

General Framework. Aggregations Procedures. Optimality in classification.

Question 1. Why there is a breakdown at h = 1 ? κ = 1 for any h > 1.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-50
SLIDE 50

General Framework. Aggregations Procedures. Optimality in classification.

Question 2 : Do we really need agg. with exp. weights ?

Theorem (suboptimality of selectors)

For any M ≥ 2, φ : R − → R s.t. φ(−1) = φ(1), ∃f1, . . . , fM : X − → {−1, 1} s.t. for any selector ˜ fn, ∃π s.t. E

  • Aφ(˜

fn) − Aφ∗ ≥ min

j=1,...,M

  • Aφ(fj) − Aφ∗

+ C

  • log M

n .

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-51
SLIDE 51

General Framework. Aggregations Procedures. Optimality in classification.

Question 2 : Do we really need agg. with exp. weights ?

Theorem (suboptimality of selectors under the margin assumption)

For any M ≥ 2, κ ≥ 1, φ : R − → R s.t. φ(−1) = φ(1), ∃f1, . . . , fM : X − → {−1, 1} s.t. for any selector ˜ fn, ∃π satisfying the φ0−MA(κ) s.t. E

  • Aφ(˜

fn) − Aφ∗ ≥ min

j=1,...,M

  • Aφ(fj) − Aφ∗

+ C log M n

  • κ

2κ−1 .

  • log M

n >> log M n

  • κ

2κ−1 >> log M

n , 1 < κ < ∞.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-52
SLIDE 52

General Framework. Aggregations Procedures. Optimality in classification.

Question 2 : Do we really need agg. with exp. weights ?

Suboptimality of Penalized ERM.

For any M ≥ 2, κ > 1 and φ : R − → R s.t. φ(−1) = φ(1), ∃f1, . . . , fM : X − → {−1, 1}, ∃π satisfying the φ0−MA(κ) s.t. the pERM aggregate ˜ f pERM

n

∈ Arg min

j=1,...,M(Aφ n(fj) + pen(fj)),

where |pen(f )| < 1

6

  • log M

n

, satisfies E

  • Aφ(˜

f pERM

n

) − Aφ∗ ≥ min

j=1,...,M

  • Aφ(fj) − Aφ∗

+ C

  • log M

n if √M log M ≤ √n/(132e3), for any integer n ≥ 1.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-53
SLIDE 53

General Framework. Aggregations Procedures. Optimality in classification.

Conclusion of optimality The margin parameter characterizes the quality of aggregation and estimation in a given model.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6

slide-54
SLIDE 54

General Framework. Aggregations Procedures. Optimality in classification.

Conclusion of optimality The margin parameter characterizes the quality of aggregation and estimation in a given model. We need convex aggregates to achieve the optimal rate of aggregation for convex losses.

Suboptimality of Penalized Empirical Risk Minimization in Classification. Universit´ e Paris 6