CO COMPSTAT2010 in Paris S 2010 Ensembled Multivariate Adaptive Regression Splines Ensembled Multivariate Adaptive Regression Splines with Nonnegative Garrote Estimator ith N ti G t E ti t Hiroki Motogaito g Osaka University Osaka University M Masashi Goto hi G t Biostatistical Research Association NPO Biostatistical Research Association, NPO. JAPAN JAPAN
Agenda Agenda g • Introduction and motivation • Introduction and motivation • Tree methods Tree methods Multivariate Adaptive Regression M lti Multivariate Adaptive Regression i t Ad ti R i Splines(MARS) Splines(MARS) p ( ) Bagging MARS Bagging MARS gg g • Our method proposed Our method proposed O th d d Ensembled MARS with nonnegative garrote Ensembled MARS with nonnegative garrote • Example and simulation • Example and simulation • Concluding remarks • Concluding remarks g 2
Agenda Agenda g • Introduction and motivation • Introduction and motivation • Tree methods Tree methods Multivariate Adaptive Regression M lti Multivariate Adaptive Regression i t Ad ti R i Splines(MARS) Splines(MARS) p ( ) Bagging MARS Bagging MARS gg g • Our method proposed Our method proposed O th d d Ensembled MARS with nonnegative garrote Ensembled MARS with nonnegative garrote • Example and simulation • Example and simulation • Concluding remarks • Concluding remarks g 3
Introduction and motivation Introduction and motivation Unstable Unstable Less interpretable Less interpretable ˆ f f ( ( x ) ) ˆ x ˆ x f f ( ( x ) ) f f ( ( x ) ) ˆ x ˆ St bili i Stabilizing f ( ) MARS Bagging gg g (Friedman,1991) (Friedman,1991) (Breiman,1996) (Breiman,1996) M ti Motivation ti a new version MARS that has both stability and interpretability i MARS th t h b th t bilit d i t t bilit 4
Agenda Agenda g • Introduction and motivation • Introduction and motivation • Tree methods Tree methods Multivariate Adaptive Regression M lti Multivariate Adaptive Regression i t Ad ti R i Splines(MARS) Splines(MARS) p ( ) Bagging MARS Bagging MARS gg g • Our method proposed Our method proposed O th d d Ensembled MARS with nonnegative garrote Ensembled MARS with nonnegative garrote • Example and simulation • Example and simulation • Concluding remarks • Concluding remarks g 5
M lti Multivariate Adaptive Regression Splines(Friedman,1991) i t Ad ti R i S li (F i d 1991) • Model form • Model form Regression model Regression model Basis function Basis function M M K K m ˆ ˆ ˆ f f m B B ( x ( ) ) q q B B ( ( x ) ) [ [ i i ( ( x t )] )] MARS 0 m m m ( ( k k , , m m ) ) p p ( ( k k , , m m ) ) ( ( k k , , m m ) ) m 1 k 1 • Algorithms • Algorithms Forward stepwise Forward stepwise 0.5 0.45 0.4 Increase basis functions Increase basis functions 0.35 Backward stepwise Backward stepwise 0.3 0 3 数の値 基底関数 0 25 0.25 P Prune off ff 基 0 2 0.2 0.15 0.15 Select the best tree Select the best tree [ [ ( ( x 0 . 5 )] )] [ [ ( ( x 0 . 5 )] )] 0.1 p p p p 0.05 1 x x 0 0 0 0.2 0 2 0.4 0 4 0 6 0.6 0 8 0.8 1 1 0 2 0.2 0.4 0.5 0.6 0 4 x 0 5 0 6 0.8 1 0 8 p p q= 1 and knot t= 0.5 1 d k t t 0 5 6
Bagging (Breiman,1996) Bagging (Breiman 1996) gg g ( , ) • Model form(Bagging MARS) • Model form(Bagging MARS) Regression model Regression model Each tree Each tree 1 1 E f ˆ ˆ ˆ f f f ( ( x ) ) f f ( ( x ) ) : MARS model MARS d l Bagging gg g MARS e e E E e e 1 1 • Algorithms • Algorithms Sample p Bootstrap sample Bootstrap sample Bootstrap sample Bootstrap sample Bootstrap sample ＋ ＋・・・＋ ＋・・・＋ ˆ ˆ ˆ ˆ f f 1 x ( ( x ) ) f f 2 x ( ( ) ) f f ( ( x ) ) f f ( x ( ) ) e E ˆ f ( x ) averaging 7 7
Agenda Agenda g • Introduction and motivation • Introduction and motivation • Previous research Pre io s research Multivariate Adaptive Regression M lti Multivariate Adaptive Regression i t Ad ti R i Splines(MARS) Splines(MARS) p ( ) Bagging MARS Bagging MARS gg g • Our method proposed Our method proposed O th d d Ensembled MARS with nonnegative garrote Ensembled MARS with nonnegative garrote • Example and simulation • Example and simulation • Concluding remarks • Concluding remarks g 8
Proposed method Proposed method p Motivation a new version MARS that has both stability and interpretability a new version MARS that has both stability and interpretability Stable, but less interpretable Stable and interpretable 2 3 1 1 1 Selection 4 & Ranking Ranking 5 5 4 4 Typical tree Typical tree nonnegative nonnegative Bagging Bagging Proposed method Proposed method garrote t (B (Breiman,1995) i 1995) 9
Ensembled MARS Ensembled MARS with non-negative garrote （ 1/2 ） ith non negati e garrote （ 1/2 ） g g • Model form • Model form Regression model g Each tree ˆ ˆ ˆ x E E ˆ c c ˆ f f c c e f f ( ( x x ) ) f f ( ( x ) ) : MARS model : MARS model , ： non-negative garrote estimator ： non negative garrote estimator e e e e 1 • Algorithms • Algorithms Generate Bagging trees Generate Bagging trees. ˆ c c Att Attach h on each tree and estimate using nonnegative h t d ti t i ti e e e e garrote(Breiman,1995). g ( , ) ˆ ― Select candidate trees(If , the tree is removed). Select candidate trees(If c c 0 0 the tree is removed) ˆ e e ˆ E ˆ f f c e f f ( ( x ) ) Get . e e 1 1 ˆ • • Interpretable structure through typical tree(max Interpretable structure through typical tree(max ) c c ) e 10 10
Ensembled MARS Ensembled MARS with non-negative garrote （ 2/2 ） ith non negati e garrote （ 2/2 ） g g non-negative garrote (Breiman,1995) ti t (B i 1995) N P P ˆ ˆ P P ( ( p ) ) 2 2 ˆ { { c } } arg g min ( ( Y c x ) ) c 0 , , c s , subject to , , j , p p 1 1 n n p p p p n n p p p p P P { c } p 1 n 1 p 1 1 p ˆ s 1 1 P P where h is the least square estimator and . i th l t ti t d p Ensembled MARS with non-negative garrote g g N N E E E E ˆ E 2 { { c c ˆ } } arg arg min min ( ( Y Y c c f f ( ( x x ) ) ) ) c c 0 0 , c c 1 1 , subject to , bj t t e 1 n e e n e e E { { c e c } } 1 1 n 1 1 e 1 1 e 1 1 ˆ ˆ f f x ) ( ( ) where is MARS model. e e n n characteristics h t i ti • All All indicates Bagging. indicates Bagging c e c 1 1 / / E E s s s s 1 • Selection of optimal is unnecessary( ). p y( ) 11 11
Agenda Agenda g • Introduction and motivation • Introduction and motivation • Previous research Pre io s research Multivariate Adaptive Regression M lti Multivariate Adaptive Regression i t Ad ti R i Splines(MARS) Splines(MARS) p ( ) Bagging MARS Bagging MARS gg g • Our method proposed O Our method proposed th d d Ensembled MARS with non-negative garrote Ensembled MARS with non negative garrote • Example and simulation • Example and simulation • Concluding remarks • Concluding remarks g 12 12
Literature example Literature example p Prostate cancer data (Stamney et al 1989: Tibshirani 1996) Prostate cancer data (Stamney et al .,1989: Tibshirani,1996) y y • : Level of prostate-specific antigen L l f t t ifi ti T T x ( ( x , , ,..., x ) ) • : Clinical measures 1 1 8 8 x : Log of tumor size x : Log of tumor size 1 x : Weight of prostate : Weight of prostate 2 2 x : Patient’s age P ti t’ 3 x : Log of benign prostatic hyperplasia amount x : Log of benign prostatic hyperplasia amount 4 4 x : Dummy variables of whether it is metastasizing to seminal vesicle y g 5 5 x : Log of capsular penetration x : Log of capsular penetration 6 x : Gleason score : Gleason score 7 7 x : Gleason score’s ratio of 4 or 5 x Gl ’ ti f 4 5 8 Sample size : 97 N • p 13 13
Literature example Literature example p 0 5 0.5 0.45 0 4 0.4 V CV 0 35 0.35 GC G 0.3 0 25 0.25 0 2 0.2 Ensembled Bagging gg g MARS NNG MARS-NNG MARS MARS 14 14
Literature example Literature example p • Number of trees • Number of trees Bagging Ensembled MARS-NNG 97 97 9 9 • Structure • Structure Bagging Ensembled MARS-NNG x x x x 2 1 2 2 x x x 4 1 Typical tree Typical tree candidates candidates 15 15
Recommend
More recommend
