BayesX: Analysing Bayesian semiparametric regression models Andreas - - PowerPoint PPT Presentation

BayesX: Analysing Bayesian semiparametric regression models

Andreas Brezger, Thomas Kneib and Stefan Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Workshop AG-Bayes, 6. Dezember 2002

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Outline of the Talk

• What is BayesX?
• Bayesian semiparametric regression
• Example(s)

BayesX: Analysing Bayesian semiparametric regression models 1

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

What is BayesX?

BayesX is a tool for Bayesian inference via MCMC simulation techniques. available as a Windows (NT, 95, 98, 2000) based application at http://www.stat.uni-muenchen.de/~lang/

BayesX: Analysing Bayesian semiparametric regression models 2

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Features of the current version

• Functions for handling and manipulating data
• Functions for handling spatial data
• Functions for drawing geographical maps,

scatterplots, etc.

• Bayesian semiparametric regression
• Model selection for DAG’s

(by Eva-Maria Fronk)

BayesX: Analysing Bayesian semiparametric regression models 3

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Features of the regression tool

• Estimation of any generalized additive model
• Response: Gaussian, Poisson, Gamma, Binomial, Multinomial

BayesX includes as special cases . . .

• Generalized linear models
• Generalized additive models
• Dynamic or state space models
• Varying coefficient models
• Mixed models
• BYM model for disease mapping

BayesX: Analysing Bayesian semiparametric regression models 4

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Observation models

• Distributional and structural assumptions, given covariates and

parameters, are based on generalized linear models.

• Response: Gaussian, Gamma, Poisson, Binomial, Multinomial
• Replace the linear predictor

η = z′γ by a semiparametric additive predictor η = f1(x1) + · · · + fp(xp) + z′γ f1, ..., fp are unknown functions of the covariates γ parameter vector for fixed effects

BayesX: Analysing Bayesian semiparametric regression models 5

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Extensions

Varying coefficient terms η = · · · + f(x)z + · · · Surface smoothing η = · · · + f(x1, x2) + · · ·

BayesX: Analysing Bayesian semiparametric regression models 6

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Priors for a function f

η = f1(x1) + · · · + fp(xp) + z′γ f = Xβ X design matrix β are unknown parameters η = · · · + Xβ + · · ·

BayesX: Analysing Bayesian semiparametric regression models 7

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

The general prior

β|τ 2 ∝ exp(− 1

2τ2β′Kβ)

τ 2 ∼ IG(a, b)

• K is a penalty matrix that penalizes too rough functions f
• structure of K depends on type of covariate and on prior beliefs on

smoothness of f

• amount of smoothness is controlled by τ 2

BayesX: Analysing Bayesian semiparametric regression models 8

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Example 1: P-splines

(Eilers and Marx, 1996; Lang and Brezger, 2002) f(x) = Spline of degree l with equally spaced inner knots ξ1, ..., ξr between x(min) and x(max) = β1B1(x) + · · · + βr+l+1Br+l+1(x) B1, ..., Br+l+1 B-spline Basis X design matrix with elements X(i, j) = Bj(xi)

BayesX: Analysing Bayesian semiparametric regression models 9

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

a) Spline vom Grad 0 .25 .5 .75 1 .5 1 1.5 b) Spline vom Grad 1 .25 .5 .75 1 .4 .45 .5 c) Spline vom Grad 2 .25 .5 .75 1 .4 .6 .8 1

BayesX: Analysing Bayesian semiparametric regression models 10

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

a) Spline vom Grad 0, B-spline Basisfunktion B^0_1 .25 .5 .75 1 .5 1 b) Spline vom Grad 0, B-spline Basisfunktion B^0_2 .25 .5 .75 1 .5 1 c) Spline vom Grad 0, B-spline Basisfunktion B^0_3 .25 .5 .75 1 .5 1 d) Spline vom Grad 0, B-spline Basisfunktion B^0_4 .25 .5 .75 1 .5 1

BayesX: Analysing Bayesian semiparametric regression models 11

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

a) Spline vom Grad 1, B-spline Basisfunktionen

• .25

.25 .5 .75 1 1.25 .5 1 b) Spline vom Grad 2, B-spline Basisfunktionen

• .5
• .25

.25 .5 .75 1 1.25 1.5 .2 .4 .6 .8

BayesX: Analysing Bayesian semiparametric regression models 12

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Example 1: P-splines, frequentist version

• relatively large number of inner knots
• difference penalty for β1, ..., βr+l+1 to penalize

too rough functions f

• Leads to penalized likelihood estimation

L = l − λ

m

• s=k+1

(∆kβs)2 ∆k denotes the difference operator of order k.

• Problem: Estimation of the smoothing parameter λ.

BayesX: Analysing Bayesian semiparametric regression models 13

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

• o o o o o o o o o o o o o o o o o o o o o

lambda=1000000

parameter number parameter estimates 5 10 15 20

• 1.0
• 0.5

0.0 0.5 1.0

lambda=1000000

covariate values function estimates

• 3
• 2
• 1

1 2 3

• 1.0
• 0.5

0.0 0.5 1.0

• o o o
• o o
• lambda=100

parameter number parameter estimates 5 10 15 20

• 1.0
• 0.5

0.0 0.5 1.0

lambda=100

covariate values function estimates

• 3
• 2
• 1

1 2 3

• 1.0
• 0.5

0.0 0.5 1.0

• o
• o
• lambda=0.001

parameter number parameter estimates 5 10 15 20

• 1.0
• 0.5

0.0 0.5 1.0

lambda=0.001

covariate values function estimates

• 3
• 2
• 1

1 2 3

• 1.0
• 0.5

0.0 0.5 1.0

BayesX: Analysing Bayesian semiparametric regression models 14

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Example 1: P-splines, Bayesian approach

• replace difference penalties by their stochastic analogues
• smoothness prior for β1, ..., βr+l+1 to penalize too rough functions f
• use first or second order random walks as smoothness prior:

βt = βt−1 + ut (RW1) βt = 2βt−1 − βt−2 + ut (RW2) ut ∼ N(0, τ 2)

BayesX: Analysing Bayesian semiparametric regression models 15

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

RW1: P(βs|βs−1, βs+1)

✻ ✲ s s s

−1 1 βs−1 βs βs+1

✻ ❄τ 2/2

BayesX: Analysing Bayesian semiparametric regression models 16

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

RW2: P(βs|βs−1, βs−2)

✻ ✲ s s s

−2 −1 βs−2 βs−1 βs

✻ ❄

τ 2 τ 2

BayesX: Analysing Bayesian semiparametric regression models 17

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

RW2: P(βs|βs−1, βs−2, βs+1, βs+2)

✻ ✲ s s s s s

• 2
• 1

1 2

βs−2, βs−1 βs βs+1 βs+2

✻ ❄τ 2/6

BayesX: Analysing Bayesian semiparametric regression models 18

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Example 2: Markov random fields

• Markov random fields (Besag, York, Mollie 1991), e.g.

βs|β−s, τ 2 ∼ N  

j∈∂s

1 Ns βj, 1 Ns τ 2   ∂s denoting the sites, that are neighbors of site s Ns number of neighbors

• X 0/1 design matrix

BayesX: Analysing Bayesian semiparametric regression models 19

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen BayesX: Analysing Bayesian semiparametric regression models 20

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Example 3: 2-dimensional surfaces

η = · · · + f1(x1) + f2(x2) + f1,2(x1, x2) + · · · f1,2 = tensor product of one dimensional B-splines =

m

• ρ=1

m

• ν=1

βρ,νB1,ρ(x1)B2,ν(x2). spatial smoothness prior fo coefficients βρ,ν, e.g. 2-dimensional random walks

BayesX: Analysing Bayesian semiparametric regression models 21

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Further examples

• random intercepts and slopes
• varying coefficient models
• time varying seasonal effects

BayesX: Analysing Bayesian semiparametric regression models 22

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Bayesian Inference via MCMC

• Draw random numbers from the posterior.
• Estimate characteristics of the posterior by their empirical analogue.
• Efficiency guaranteed by matrix operations for sparse matrices.

Details in Fahrmeir, Lang (2001a,b) Lang and Brezger (2002)

BayesX: Analysing Bayesian semiparametric regression models 23

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Example: Gaussian response

yi = f1(xi1) + f2(xi2) + ǫi y = X1β1 + X2β2 + ǫ

• Draw random number from β1|· which is Gaussian with

Σ1 =

• X′

1X1

σ2

+ K1

τ2

1

−1 µ1 = 1

σ2Σ1X′ 1(y − X2β2)

• Draw random number from β2|· which is Gaussian with

Σ2 =

• X′

2X2

σ2

+ K2

τ2

2

−1 µ2 = 1

σ2Σ2X′ 2(y − X1β1)

• Draw random numbers from full conditionals of variance parameters

σ2|·, τ 2

1|·,

τ 2

2|·.

BayesX: Analysing Bayesian semiparametric regression models 24

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

breite hoehe

BayesX: Analysing Bayesian semiparametric regression models 25

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Application : ”rental guides” for flats

Response variable R = monthly rent per square meter in German Marks Covariates F floor space in square meters A age of building, that is year of construction L location in subquarters of the building in Munich z vector of categorical covariates characterizing the flat

BayesX: Analysing Bayesian semiparametric regression models 26

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Model 1: η = γ0 + f1(F) + f2(A) + f12(F, A) + bunstr

L

+ bstr

L + z′γ

f1, f2 P-spline of degree 3 with second order random walk penalty f12 tensor product P-spline with 2-dim. first order random walk penalty bunstr

L

uncorrelated random effect bstr

L

spatially correlated random effect, MRF Model 2: η = γ0 + f1(F) + f2(A) + f12(F, A) + bunstr

L

+ bstr

L + γ1 gL + γ2 tL + z′γ

where gL and tL are 0/1 indicators for good location and top location assessed by experts.

BayesX: Analysing Bayesian semiparametric regression models 27

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

a) Main effect of floor space floor space in square meters 50 100 150

• 5

5 10 b) Main effect of year of construction year of construction 1920 1940 1960 1980 2000

• 1

1 2 3

BayesX: Analysing Bayesian semiparametric regression models 28

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen BayesX: Analysing Bayesian semiparametric regression models 29

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

• 1.7

1.7

a) Experts assessment excluded

• 1.7

1.7

b) Experts assessment included

BayesX: Analysing Bayesian semiparametric regression models 30

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Example: Forest health data

Variable of primary interest yit, the degree of defoliation of beech i in year t, measured in three catogories: yit = 1 no defoliation yit = 2 defoliation 25% or less yit = 3 defoliation above 25% Covariates T Calendar time in years (1983-2001) S site of the beech (84 sites) A age of the tree in years (7-231) C Canopy density at the stand measured in percentages 0%,10%,...,90%,100%

BayesX: Analysing Bayesian semiparametric regression models 31

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Model Cumulative probit model, i.e. Uit = f1(t) + f2(Ait) + f3(Cit) + fspat(si) + z′

itγ + ǫit

BayesX: Analysing Bayesian semiparametric regression models 32

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

1983 1987 1992 1996 2001

• 1.58
• 0.93
• 0.27

0.37 1.02 Year 7 63 119 175 231

• 6.64
• 3.70
• 0.76

2.18 5.12 Age of the tree

BayesX: Analysing Bayesian semiparametric regression models 33

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

0.25 0.5 0.75 1

• 1.68
• 0.82

0.04 0.91 1.77 Canopy density

BayesX: Analysing Bayesian semiparametric regression models 34

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Figure 1: Posterior probabilities of the spatial effect for a nominal level of 80%. black=positive effect,white=negative effect

BayesX: Analysing Bayesian semiparametric regression models 35

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

• 1.0

1.0

Figure 2: Posterior probabilities of the spatial effect for a nominal level of 95%. black=positive effect,white=negative effect

BayesX: Analysing Bayesian semiparametric regression models 36

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Density

Kernel Density Estimate

• 5

5 .1 .2 .3

Figure 3: Kernel density estimator of the posterior mean of the spatial effect

BayesX: Analysing Bayesian semiparametric regression models 37

• A. Brezger, T. Kneib and S. Lang

Institut f¨ ur Statistik, Universit¨ at M¨ unchen

Classification table yit ˆ yit 1 2 3 1 896 74 1 2 116 366 11 3 40 45 Missclassified: 15.5% Classification table if spatial effect is not considered yit ˆ yit 1 2 3 1 860 111 2 233 256 4 3 12 69 4 Missclassified: 27.6%

BayesX: Analysing Bayesian semiparametric regression models 38