[PPT] - Boosting: more than an ensemble method for prediction Peter B PowerPoint Presentation

SLIDE 1

✬ ✫ ✩ ✪ Boosting: more than an ensemble method for prediction

Peter B¨ uhlmann ETH Z¨ urich

1

SLIDE 2

✬ ✫ ✩ ✪

1. Historically: Boosting is about multiple predictions

Data: (X1, Y1), . . . , (Xn, Yn) (i.i.d. or stationary), predictor variables Xi ∈ Rp response variables Yi ∈ R or Yi ∈ {0, 1, . . . , J − 1} Aim: estimation of function f(·) : Rp → R, e.g.

f(x) = I E[Y |X = x] or f(x) = I P[Y = 1|X = x] with Y ∈ {0, 1}

r distribution of survival time Y given X depends on some function f(X) only

“historical” view (for classification): Boosting is a multiple predictions (estimation) & combination method

2

SLIDE 3

✬ ✫ ✩ ✪

Base procedure:

data

algorithm A

− →

ˆ θ(·) (a function estimate)

e.g.: simple linear regression, tree, MARS, “classical” smoothing, neural nets, ... Generating multiple predictions: weighted data 1 algorithm A

− → ˆ θ1(·)

weighted data 2 algorithm A

− → ˆ θ2(·) · · · · · ·

weighted data M algorithm A

− → ˆ θM(·)

Aggregation: ˆ

fA(·) = M

m=1 amˆ

θm(·)

data weights? averaging weights am?

3

SLIDE 4

✬ ✫ ✩ ✪

classification of 2 lymph nodal status in breast cancer using gene expressions from microarray data:

n = 33, p = 7129 (for CART: gene-preselection, reducing to p = 50)

method test set error gain over CART CART 22.5% – LogitBoost with trees 16.3% 28% LogitBoost with bagged trees 12.2% 46% this kind of boosting: mainly prediction, not much interpretation

4

SLIDE 5

✬ ✫ ✩ ✪

2. Boosting algorithms

around 1990: Schapire constructed some early versions of boosting AdaBoost proposed for classification by Freund & Schapire (1996) data weights (rough original idea): large weights to previously heavily misclassified instances (sequential algorithm) averaging weights am: large if in-sample performance in mth round was good Why should this be good?

5

SLIDE 6

✬ ✫ ✩ ✪

Why should this be good? some common answers 5 years ago ... because

it works so well for prediction (which is quite true)
it concentrates on the “hard cases” (so what?)
AdaBoost almost never overfits the data no matter how many iterations it is run

(not true)

6

SLIDE 7

✬ ✫ ✩ ✪

A better explanation Breiman (1998/99): AdaBoost is functional gradient descent (FGD) procedure aim: find f ∗(·) = argminf(·)I

E[ρ(Y, f(X))]

e.g. for ρ(y, f) = |y − f|2 f ∗(x) = I

E[Y |X = x]

FGD solution: consider empirical risk n−1 n

i=1 ρ(Yi, f(Xi)) and

do iterative steepest descent in function space

7

SLIDE 8

✬ ✫ ✩ ✪ 2.1. Generic FGD algorithm

Step 1. ˆ

f0 ≡ 0; set m = 0.

Step 2. Increase m by 1. Compute negative gradient − ∂

∂f ρ(Y, f)

and evaluate at f = ˆ

fm−1(Xi) = Ui (i = 1, . . . , n)

Step 3. Fit negative gradient vector U1, . . . , Un by base procedure

(Xi, Ui)n

i=1

algorithm A

− → ˆ θm(·)

e.g. ˆ

θm fitted by (weighted) least squares

i.e. ˆ

θm(·) is an approximation of the negative gradient vector

Step 4. Up-date ˆ

fm = ˆ fm−1(·) + νsm · ˆ θm(·) sm = argminsn−1 n

i=1 ρ(Yi, ˆ

fm−1(Xi) + s · ˆ θm(Xi)) and 0 < ν ≤ 1

i.e. proceed along an estimate of the negative gradient vector Step 5. Iterate Steps 2-4 until m = mstop for some stopping iteration mstop

8

SLIDE 9

✬ ✫ ✩ ✪

Why “functional gradient”? Alternative formulation in function space: empirical risk functional: C(f) = n−1 n

i=1 ρ(Yi, f(Xi))

inner product:

f, g = n−1 n

i=1 f(Xi)g(Xi)

negative Gateaux derivative:

−dC(f)(x) = ∂ ∂αC(f + α1x)|α=0, −dC( ˆ fm−1)(Xi) = Ui

if U1, ..., Un are fitted by least squares: equivalent to maximize −dC(fm), θ w.r.t. θ(·) (ifθ = 1) (over all possible θ(·)’s from the base procedure) i.e: ˆ

θm(·) is the best approximation (most parallel)

to the negative gradient −dC(fm)

9

SLIDE 10

✬ ✫ ✩ ✪

By definition: FGD yields additive combination of base procedure fits

ν mstop

m=1 smˆ

θm(·)

Breiman (1998): FGD with ρ(y, f) = exp((2y − 1) · f) for binary classification yields the AdaBoost algorithm (great result!) Remark: FGD can not be represented as some explicit estimation function(al):

ˆ fm(·)=argminf∈Fn−1

n

i=1

ρ(Yi, f(Xi))

for some function class F

FGD is mathematically more difficult to analyze but

generically applicable (as an algorithm!) in very complex models

10

SLIDE 11

✬ ✫ ✩ ✪ 2.2. L2Boosting

(see also Friedman, 2001) loss function ρ(y, f) = |y − f|2 population minimizer: f ∗(x) = I

E[Y |X = x]

FGD with base procedure ˆ

θ(·): repeated fitting of residuals m = 1 : (Xi, Yi)n

i=1 ˆ

θ1(·), ˆ f1 = νˆ θ1

resid. Ui = Yi − ˆ

f1(Xi) m = 2 : (Xi, Ui)n

i=1 ˆ

θ2(·), ˆ f2 = ˆ f1 + νˆ θ2

resid. Ui = Yi − ˆ

f2(Xi) ... ... ˆ fmstop(·) = ν mstop

m=1 ˆ

θm(·) (stagewise greedy fitting of residuals)

Tukey (1977): twicing for mstop = 2 and ν = 1

11

SLIDE 12

✬ ✫ ✩ ✪

Any gain over classical methods? (for additive modeling)

Ozone data: n=300, p=8

boosting iterations MSE 20 40 60 80 100 18 19 20 21 22

n = 300, p = 8

magenta: L2Boosting with stumps

(horiz. line = cross-validated stopping)

black: L2Boosting with componentwise

smoothing spline (horiz. line = cross-validated stopping) i.e: smoothing spline fi tting against the selected predictor which reduces RSS most

green: MARS restricted to additive modeling
red: additive model using backfi tting

L2Boosting with stumps or comp. smoothing splines also yields additive model: mstop

m=0 ˆ

θm(x( ˆ

Sm)) = ˆ

g1(x(1)) + . . . + ˆ gp(x(p))

12

SLIDE 13

✬ ✫ ✩ ✪

Simulated data: non-additive regression function, n = 200, p = 100

Regression: n=200, p=100

boosting iterations MSE 50 100 150 200 250 300 11 12 13 14 15 16

magenta: L2Boosting with stumps
black: L2Boosting with componentwise
green: MARS restricted to additive modeling
red: additive model using backfi tting and
fwd. var. selection

13

SLIDE 14

✬ ✫ ✩ ✪

similar for classifi cation

14

SLIDE 15

✬ ✫ ✩ ✪

3. Structured models and choosing the base procedure

have just seen the Componentwise smoothing spline base procedure smoothes the reponse against the one predictor variable which reduces RSS most we keep the degrees of freedom fixed for all candidate predictors, e.g. d.f. = 2.5

L2Boosting yields an additive model fit, including variable selection

15

SLIDE 16

✬ ✫ ✩ ✪

Componentwise linear least squares simple linear OLS against the one predictor variable which reduces RSS most

ˆ θ(x) = ˆ β ˆ S x( ˆ S), ˆ βj = n

X

i=1 YiX(j) i / n

X

i=1 (X(j) i )2, ˆ S = arg min j n

X

i=1 (Yi − ˆ βj X(j) i )2

first round of estimation: selected predictor variable X ( ˆ

S1) (e.g. = X(3))

corresponding ˆ

β ˆ

S1 fitted function ˆ

f1(x)

second round of estimation: selected predictor variable X ( ˆ

S2) (e.g.= X(21))

corresponding ˆ

β ˆ

S2 fitted function ˆ

f2(x)

etc.

L2Boosting: ˆ fm(x) = ˆ fm−1(x) + ν · ˆ θ(x) L2Boosting yields linear model fit, including variable selection,

i.e. structured model fit

16

SLIDE 17

✬ ✫ ✩ ✪

for ν = 1, this is known as Matching Pursuit (Mallat and Zhang, 1993) Weak greedy algorithm (deVore & Temlyakov, 1997) a version of Boosting (Schapire, 1992; Freund & Schapire, 1996) Gauss-Southwell algorithm C.F . Gauss in 1803 “Princeps Mathematicorum” R.V. Southwell in 1933 Professor in engineering, Oxford

17

SLIDE 18

✬ ✫ ✩ ✪

binary lymph node classification in breast cancer using gene expressions: a high noise problem

n = 49 samples, p = 7129 gene expressions

L2Boosting

FPLR Pelora 1-NN DLDA SVM CV-misclassif.err. 17.7% 35.25% 27.8% 43.25% 36.12% 36.88%

multivariate gene selection best 200 genes from Wilcox.

L2Boosting selected 42 out of p = 7129 genes

for this data-set: not good prediction, with any of the methods but L2Boosting may be a reasonable(?) multivariate gene selection method

18

SLIDE 19

✬ ✫ ✩ ✪

42 (out of 7129) selected genes (n = 49)

10 20 30 40 −0.15 −0.10 −0.05 0.00 0.05

sorted regression coefficients

selected genes

identifiability problem: strong correlations among some genes

consider groups of highly correlated genes, biological categories (e.g. GO), ....

linear model: multivariate association between genes and tumor-type very different from 2-sample tests for individual genes

19

SLIDE 20

✬ ✫ ✩ ✪

Pairwise smoothing splines smoothes response against the pair of predictor variables which reduces RSS most we keep the degrees of freedom fixed for all candidate pairs, e.g. d.f. = 2.5

L2Boosting yields a nonparametric interaction model, including variable

selection

20

SLIDE 21

✬ ✫ ✩ ✪

Example: degree 2 nonparametric interaction modelling Friedman #1 model:

Y = 10 sin(πX1X2) + 20(X3 − 0.5)2 + 10X4 + 5X5 + N (0, 1), X = (X1, . . . , X20) ∼ Unif.([0, 1]20)

100 200 300 400 500 4 5 6 7

p=20, p−eff=10, n=50

boosting iterations MSE

MARS L2Boost

AIC_c stopped

L2Boosting with pairwise splines

sample size n = 50

p = 20, effective peff = 5

21

SLIDE 22

✬ ✫ ✩ ✪

Regression trees stumps (2 terminal nodes): L2Boosting fits an additive model trees with d terminal nodes: L2Boosting fits an interaction model of degree d − 2

22

SLIDE 23

✬ ✫ ✩ ✪

The low variance high bias “principle”

nce we have decided about some structural properties

choose base procedure with low variance but potentially large estimation bias bias can be reduced by further boosting iterations (which will increase variance) example: low degrees of freedom in componentwise smoothing splines for additive modeling a justification will be given later

23

SLIDE 24

✬ ✫ ✩ ✪

4. More on L2Boosting

L2Boosting for linear models

use componentwise linear least squares base procedure

L2Boosting converges to a least squares solution as boosting iterations m → ∞

(the unique LS solution if design has full rank p ≤ n) when stopping early:

it does variable selection
coefficient estimates are typically shrunken version of LS

“similar to” the Lasso

24

SLIDE 25

✬ ✫ ✩ ✪

Connections to Lasso (for linear models): Efron, Hastie, Johnstone, Tibshirani (2004): for special design matrices, iterations of L2Boosting with “infinitesimally” small ν yield all Lasso solutions when varying λ

computationally interesting to produce all Lasso solutions in

ne sweep of boosting

Least Angle Regression LARS (Efron et al., 2004) is computationally even more clever and efficient than L2Boosting Zhao and Yu (2005): in “general”, when adding some backward steps the solutions from Lasso and modified Boosting “coincide” greedy (plus backward steps) and convex optimization are surprisingly similar

25

SLIDE 26

✬ ✫ ✩ ✪ p = 10, peff = 3, n = 20

100 200 300 400 500 2 3 4 5 6

uncorrelated design

boosting iterations MSE AIC−stopped L2Boost Lasso fwd.var.sel. OLS 100 200 300 400 500 2 3 4 5 6

correlated design

boosting iterations MSE AIC−stopped L2Boost Lasso fwd.var.sel. OLS

26

SLIDE 27

✬ ✫ ✩ ✪

binary lymph node classification using gene expressions

n = 49 samples, p = 7129 gene expressions

L2Boosting

FPLR Pelora 1-NN DLDA SVM CV-misclassif.err. 17.7% 35.25% 27.8% 43.25% 36.12% 36.88% Lasso CV-misclassif.err. 21.2%

multivariate gene selection best 200 genes from Wilcox.

L2Boosting selected 42 out of p = 7129 genes

Lasso selected 15 genes

27

SLIDE 28

✬ ✫ ✩ ✪

how well can we do? statistically consistent for very high-dimensional, sparse linear models

Yi = β0 +

p

j=1

βjX(j)

i

+ εi (i = 1, . . . , n), p ≫ n

Theorem (PB, 2004)

L2Boosting with comp. linear LS is consistent (for suitable number of boosting

iterations) if:

pn = O(exp(Cn1−ξ)) (0 < ξ < 1)

(high-dimensional) essentially exponentially many variables relative to n

supn

pn

j=1 |βj,n| < ∞ ℓ1-sparseness of true function

i.e. for suitable, slowly growing m = mn:

I EX| ˆ fmn,n(X) − fn(X)|2 = oP (1) (n → ∞)

“no” assumptions about the predictor variables/design matrix

28

SLIDE 29

✬ ✫ ✩ ✪

analogous results also for

multivariate regression
vector autoregressive time series

(Lutz & PB, 2005)

29

SLIDE 30

✬ ✫ ✩ ✪

For linear models: L2Boosting or Lasso?

“similar” prediction performance
LARS algorithm is computationally more efficient (for all Lasso solutions)

O(np min(n, p)) for LARS; O(npmstop) for L2Boosting

notion of degrees of freedom is easier for L2Boosting
boosting is more generic (nonparametric models, other loss functions,...)

30

SLIDE 31

✬ ✫ ✩ ✪ 4.1. Degrees of freedom for boosting

(PB, 2004) the only tuning parameter: number of boosting iterations could use cross-validation works reasonably well alternatively: use AIC, BIC or gMDL as model selection criteria which involve degrees of freedom of boosting

31

SLIDE 32

✬ ✫ ✩ ✪

hat-matrix of comp.wise linear LS base procedure:

H(j) : (Y1, . . . , Yn) → ( ˆ Y1, . . . , ˆ Yn) when using the jth predictor variable only: H(j) = X(j)(X(j))T /X(j)2 L2Boosting hat-matrix: Bm = Bm−1 + ν · H( ˆ

Sm)(I − Bm−1)

= I − (I − ν · H( ˆ

Sm)

selected in mth iter.

)(I − ν · H( ˆ

Sm−1)) · · · (I − ν · H( ˆ S1))

degrees of freedom of boosting in iteration m:

d.f.(Bm) = trace(Bm) d.f. ignores the selection effect, i.e. “slightly” too small

(“negligible” (?) since we can allow for o(exp(n)) candidate basis functions)

32

SLIDE 33

✬ ✫ ✩ ✪ d.f. is very different from the number of variables in the model

example: 3 (or more) correlated variables, ν = 1 sequence of selected variables: 3,2,1,3,2,1 d.f.(B6) = 1.79 < 3 sequence of selected variables: 1,2,3,2,3,1 d.f.(B6) = 1.54 < 3

33

SLIDE 34

✬ ✫ ✩ ✪

Stopping the boosting iterations we often use the corrected AICc criterion: AICc(Bm) = log(RSSm/n) +

1 + trace(Bm)/n 1 − (trace(Bm) + 2)/n

estimate stopping iteration by

ˆ mstop = argminm AICc(Bm)

34

SLIDE 35

✬ ✫ ✩ ✪ p = 10, peff = 3, n = 20

100 200 300 400 500 2 3 4 5 6

uncorrelated design

boosting iterations MSE AIC−stopped L2Boost Lasso fwd.var.sel. OLS 100 200 300 400 500 2 3 4 5 6

correlated design

boosting iterations MSE AIC−stopped L2Boost Lasso fwd.var.sel. OLS

35

SLIDE 36

✬ ✫ ✩ ✪

Analogously for nonparametric base procedures hat-matrix H(S) with a selected subset S of predictor variables

Bm = I − (I − ν · H( ˆ

Sm))(I − ν · H( ˆ Sm−1)) · · · (I − ν · H( ˆ S1))

e.g. L2Boosting with pairwise splines for nonparametric interaction modeling

100 200 300 400 500 4 5 6 7

p=20, p−eff=10, n=50

boosting iterations MSE

MARS L2Boost

AIC_c stopped

36

SLIDE 37

✬ ✫ ✩ ✪

More on degrees of freedom example: L2Boosting with componentwise smoothing splines for additive modeling boosting hat-matrix Bm: since ˆ

f(Xi) = p

j=1 ˆ

fj(Xi) decompose Bm =

p

j=1

A(j)

m

hat-matrix for ˆ

fj(·)

easy to compute recursively:

A(j)

m = A(j) m−1 + δj, ˆ Smν · H( ˆ Sm)(I − Bm−1)

thus

d.f.

trace(Bm)

=

p

j=1

d.f.(j)

trace(A(j)

m )

37

SLIDE 38