Ensembled Multivariate Adaptive Regression Splines Ensembled - - PowerPoint PPT Presentation
CO COMPSTAT2010 in Paris S 2010 Ensembled Multivariate Adaptive Regression Splines Ensembled Multivariate Adaptive Regression Splines with Nonnegative Garrote Estimator ith N ti G t E ti t Hiroki Motogaito g Osaka University Osaka
CO S 2010 COMPSTAT2010 in Paris
Ensembled Multivariate Adaptive Regression Splines Ensembled Multivariate Adaptive Regression Splines ith N ti G t E ti t with Nonnegative Garrote Estimator
Hiroki Motogaito g Osaka University Osaka University M hi G t Masashi Goto Biostatistical Research Association NPO Biostatistical Research Association, NPO. JAPAN JAPAN
2
3
Unstable Less interpretable Unstable Less interpretable
) ( ˆ f ) (x f ˆ ) ( ˆ x f
) ( ˆ x f
St bili i
) ( ˆ x f ) (x f
) (x f
Stabilizing
Bagging MARS gg g (Breiman,1996) (Friedman,1991) (Breiman,1996) (Friedman,1991)
M ti ti Motivation
i MARS th t h b th t bilit d i t t bilit a new version MARS that has both stability and interpretability
4
5
M lti i t Ad ti R i S li (F i d 1991) Multivariate Adaptive Regression Splines(Friedman,1991)
Regression model Basis function
M K
Regression model Basis function
M
B f ) ( ˆ ˆ ˆ
m
K q
i B )] ( [ ) (
m mB
f
MARS
) (x
q m k m k p m k m
t x i B
) , ( ) , ( ) , (
)] ( [ ) (x
m 1
k m k m k p m k m 1 ) , ( ) , ( ) , (
0.5
0.45
Increase basis functions
0.4
Increase basis functions
0 3 0.35
0 25 0.3
数の値
P ff
0 2 0.25
基底関数
Prune off
0.15 0.2
基
Select the best tree
0.1 0.15
)] 5 . ( [
p
x
)] 5 . ( [
p
x
Select the best tree
0.05
)] ( [
p
)] ( [
p
0 2 0 4 0 6 0 8 1
0 2 0 4 0 5 0 6 0 8 1 x
0.2 0.4 0.6 0.8 1
p
x
0.2 0.4 0.5 0.6 0.8 1
p
x
1 d k t t 0 5 q=1 and knot t=0.5 6
Regression model Each tree Regression model
1
Each tree
) ( ˆ 1 ˆ
E f
f
MARS d l
) ( ˆ f ) (
1 MARS Bagging
x
e e
f E f
: MARS model
) (x
e
f
1 gg g
e
E
Sample p
Bootstrap sample Bootstrap sample Bootstrap sample Bootstrap sample Bootstrap sample
＋ ＋・・・＋ ＋・・・＋
) ( ˆ x f ) ( ˆ f ) ( ˆ f ) ( ˆ f ) (
1 x
f ) (
2 x
f
) ( x
e
f ) ( x
E
f
ˆ
7
averaging
) (x f
7
8
Motivation
a new version MARS that has both stability and interpretability a new version MARS that has both stability and interpretability
Stable, but less interpretable Stable and interpretable
2 3
1
Selection
1 4
1
& Ranking Ranking
4 5
Typical tree
4 5
Typical tree
nonnegative
Bagging Proposed method
nonnegative t
Bagging Proposed method
garrote (B i 1995) (Breiman,1995)
9
Regression model Each tree g
E
) ( ˆ ˆ ˆ x
E
f c f
: MARS model ：non-negative garrote estimator
) ( ˆ x f c ˆ ) (
1
x
e e e f
c f
: MARS model , ：non negative garrote estimator
) (x
e
f
e
c
h h t d ti t i ti
ˆ
e
c
e
c
garrote(Breiman,1995).
e e
g ( , ) Select candidate trees(If the tree is removed)
ˆ c
― Select candidate trees(If , the tree is removed).
e
c
) ( ˆ ˆ ˆ
E
f f
) (
1
x
e e e f
c f
)
1
c ˆ
e
c
10 10
ti t (B i 1995) non-negative garrote (Breiman,1995)
N P P 2 ) (
ˆ
P
p n p p n P p
x c Y c
P
2 ) ( 1
) ˆ ( min arg } ˆ { s c c
p p
,
, subject to ,
n p n p p n c p
P p
1 1 } { 1
) ( g } {
1
p p p
1
,
, j , h i th l t ti t d
ˆ P 1
where is the least square estimator and .
p
P s 1
Ensembled MARS with non-negative garrote g g
N E E
N E E
f c Y c
2
) ) ( ˆ ( min arg } ˆ { x 1
E
c c
bj t t
n e e n c e
f c Y c
E
1 1 } { 1
) ) ( ( min arg } {
1
x 1 ,
1
e e
c c
, subject to ,
n e ce 1 1 } {
1
1 e
ˆ
where is MARS model.
) ( ˆ
n e
f x ) (
n e
f
h t i ti characteristics
indicates Bagging
E c / 1
All indicates Bagging.
E ce / 1
s 1 s
p y( )
s s
11 11
12 12
Prostate cancer data (Stamney et al 1989: Tibshirani 1996) Prostate cancer data (Stamney et al.,1989: Tibshirani,1996)
L l f t t ifi ti
y
y
T
T 8 1
) ,..., ( x x x
: Log of tumor size
8 1
) , , ( x : Log of tumor size
1
x
: Weight of prostate
2
x : Weight of prostate
P ti t’
2
: Patient’s age
3
x
: Log of benign prostatic hyperplasia amount
4
x : Log of benign prostatic hyperplasia amount
4
x
: Dummy variables of whether it is metastasizing to seminal vesicle
5
x
y g : Log of capsular penetration
5
x : Log of capsular penetration
6
x
: Gleason score
7
x : Gleason score
Gl ’ ti f 4 5
7
x : Gleason score’s ratio of 4 or 5
8
x
N
p
13 13
0 5 0.5 0.45 0 4 0.4
V
0 35
CV
0.35
GC G
0.3 0 25 0.25 0 2 0.2
Ensembled Bagging MARS NNG gg g MARS MARS-NNG MARS
14 14
Bagging Ensembled MARS-NNG
97 9 97 9
Bagging Ensembled MARS-NNG
1
x
2
x
2
x
2
x x
1
x
4
x
Typical tree candidates Typical tree candidates 15 15
5 10 ) 5 ( 20 ) sin( 10
2
x x x x x y
where is
) 1 ( N
, 5 10 ) 5 . ( 20 ) sin( 10
5 4 3 2 1
x x x x x y
where is .
) 1 , ( N
i i l i 100
1 000
1,000
i MARS E bl d MARS NNG
(St d di d )
( q )
16 16
0.07 0.065
D
EST
0 06
MSE
0.06
M
0 055 0.055 0.05
Ensembled se b ed MARS-NNG Bagging MARS MARS MARS NNG
Number 11 6 Number
11.6
( d)
100 1
(averaged) 17 17
18 18
h d d i bl d i bl
p p p
19 19
, ( ) g g g
g , ，
140 140. B i L F i d J H Ol h R A & St C J (1984)
Cl ifi ti A d R i T W d th Classification And Regression Trees. Wadsworth.
Friedman, J. H. (1991) Multivariate Adaptive Regression Splines (with discussion) Annals of Statistics 19 1-141 (with discussion). Annals of Statistics,19, 1 141.
boosting machine Ann Statist 29(5) 1189 1232 boosting machine. Ann. Statist., 29(5), 1189-1232.
Meinshausen, N. (2009): Node Harvest: simple and interpretable regression and classification Arxiv preprint arXiv:0910 2145 regression and classification. Arxiv preprint arXiv:0910.2145.
Motogaito, H., Sugimoto, T. & Goto, M. (2007): Multivariate Adaptive Regression Splines with Non negative Garrote Estimator Japanese Regression Splines with Non-negative Garrote Estimator. Japanese J A l St ti t 36 99 118 (i J )
R St ti t S B 69(2) 143 161
20 20
21 21
22 22
0 0662
0 07
0.0597 0.0662
0.07
0 0595
0 065
0.0595
0.065
0 0545 0,0545
0 06
0.0567
0.06
0 0 44 0.0544
0 055 0.055
0 0609 0.0609
0.05
0.0573 0.0570 0.0573 0.0570
Ensembled Ensembled MARS-NNG Bagging MARS MARS MARS-NNG gg g
23 23
1
x x x
1
x
1
x
8
x
8
x
1
x
1
x x x
2
x
3
x
6
x
24 24 24 24
x
1
x
2
x
2
x
2
x
1
x x
1
x
4
x
25 25 25 25