Introduction to General and Generalized Linear Models Mixed effects - PowerPoint PPT Presentation

Introduction to General and Generalized Linear Models Mixed effects models - Part II Henrik Madsen Poul Thyregod Informatics and Mathematical Modelling Technical University of Denmark DK-2800 Kgs. Lyngby January 2011 Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 1 / 29

This lecture One-way random effects model, continued More examples of hierarchical variation General linear mixed effects models Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 2 / 29

One-way random effects model Estimation of parameters Confidence interval for the variance ratio In the balanced case one may construct a confidence interval for the variance ratio γ . A 1 − α confidence interval for γ , i.e. an interval ( γ L , γ U ) , satisfying P [ γ L < γ < γ U ] = 1 − α is obtained by using � � γ L = 1 Z − 1 n F ( k − 1 , N − k ) 1 − α/ 2 � � γ U = 1 Z − 1 n F ( k − 1 , N − k ) α/ 2 Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 3 / 29

One-way random effects model Estimation of parameters Theorem (Moment estimates in the random effects model) Moment estimates for the parameters µ, σ 2 and σ 2 u are � µ = Y ·· σ 2 = SSE / ( N − k ) � σ 2 2 = SSB / ( k − 1) − SSE / ( N − k ) = SSB / ( k − 1) − � σ u � n 0 n 0 where the weighted average group size n 0 is given by �� k � � k i / � k = N − � 1 n 2 1 n i − 1 n i i n 2 i /N n 0 = k − 1 k − 1 Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 4 / 29

One-way random effects model Estimation of parameters Distribution of “residual” sum of squares In the balanced case we have that SSE ∼ σ 2 χ 2 ( k ( n − 1)) SSB ∼ { σ 2 /w ( γ ) } χ 2 ( k − 1) and that SSE and SSB are independent. Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 5 / 29

One-way random effects model Estimation of parameters Unbiased estimates for variance ratio in the balanced case In the balanced case, n 1 = n 2 = · · · = n k = n , we can provide explicit unbiased estimators for γ and w ( γ ) = 1 / (1 + nγ ) . One has w = SSE / { k ( n − 1) } � SSB / ( k − 3) � � γ = 1 SSB / ( k − 1) � SSE / { k ( n − 1) − 2 } − 1 n are unbiased estimators for w ( γ ) = 1 / (1 + nγ ) and for γ = σ 2 u /σ 2 , respectively. Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 6 / 29

One-way random effects model Example - Wool data s 2 = SS/f E[ S 2 ] Variation Sum of squares f σ 2 + 4 σ 2 Between bales SSB 65.9628 6 10.9938 u σ 2 Within bales SSE 131.4726 21 6.2606 Table: ANOVA table for the baled wool data. Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 7 / 29

One-way random effects model Example - Wool data The test statistic for the hypothesis H 0 : σ 2 u = 0 , is z = 10 . 9938 6 . 2606 = 1 . 76 < F 0 . 95 (6 , 21) = 2 . 57 The p -value is P [ F (6 , 21) ≥ 1 . 76] = 0 . 16 Thus, the test fails to reject the hypothesis of no variation between the purity of the bales when testing at a 5% significance level. However, as the purpose is to describe the variation in the shipment, we will estimate the parameters in the random effects model, irrespective of the test result. Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 8 / 29

One-way random effects model Example - Wool data Now lets find a 95% confidence interval for the ratio γ = σ 2 u /σ 2 . As F (6 , 21) 0 . 025 = 1 /F (21 , 6) 0 . 975 , one finds the interval � � � 1 . 76 � γ L = 1 1 . 76 − 1 = 0 . 25 × 3 . 09 − 1 = − 0 . 11 4 F (6 , 21) 0 . 975 � � γ U = 1 1 . 76 − 1 = 0 . 25 × (1 . 76 × 5 . 15 − 1) = 2 . 02 4 F (6 , 21) 0 . 025 Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 9 / 29

One-way random effects model Maximum likelihood estimates Theorem (Maximum likelihood estimates for the parameters under the random effects model) The maximum likelihood estimates for µ, σ 2 and σ 2 u = σ 2 γ are determined by For � i ( y i · − y ·· ) 2 < SSE + SSB one obtains i n 2 � µ = y ·· = 1 � n i y i · N i σ 2 = 1 � N (SSE + SSB) � γ = 0 Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 10 / 29

One-way random effects model Maximum likelihood estimates Theorem (Maximum likelihood estimates for the parameters under the random effects model continued) For � i ( y i · − y ·· ) 2 > SSE + SSB the estimates are determined as i n 2 solution to k � 1 µ = � n i w i ( � γ ) y i · W ( � γ ) i =1 � � k � σ 2 = 1 µ ) 2 � SSE + n i w i ( � γ )( y i · − � N i =1 � � k k � � 1 µ ) 2 = 1 n 2 γ ) 2 ( y i · − � µ ) 2 i w i ( � SSE + n i w i ( � γ )( y i · − � W ( � γ ) N i =1 i =1 where k � W ( γ ) = n i w i ( γ ) . i =1 Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 11 / 29

One-way random effects model Maximum likelihood estimates The maximum likelihood estimate � µ is a weighted average of the group averages µ is a weighted average of the group averages, y i · , with the estimates for � the marginal precisions σ 2 σ 2 n i w i ( γ ) = Var[ Y i · ] as weights. We have the marginal variances u + σ 2 = σ 2 σ 2 Var[ Y i · ] = σ 2 (1 + n i γ ) = n i n i n i w i ( γ ) When the experiment is balanced , i.e. when n 1 = n 2 = · · · = n k , then all weights are equal, and one obtains the simple result that � µ is the crude average of the group averages. Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 12 / 29

One-way random effects model Maximum likelihood estimates The estimate for σ 2 utilizes also the variation between groups We observe that the estimate for σ 2 is not only based upon the variation within groups, SSE , but the estimate does also utilize the knowledge of the variation between groups, as σ 2 E[( Y i · − µ ) 2 ] = Var[ Y i · ] = n i w i ( γ ) and therefore, the terms ( y i · − µ ) 2 contain information about σ 2 as well as γ . Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 13 / 29

One-way random effects model Maximum likelihood estimates The estimate for σ 2 is not necessarily unbiased We observe further that – as usual with ML-estimates of variance – the estimate for σ 2 is not necessarily unbiased. Instead of the maximum likelihood estimate above, it is common practice to adjust the estimate. Later we shall introduce the so-called residual maximum likelihood (REML) estimates for σ 2 and σ 2 u , obtained by considering the distribution of the residuals. Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 14 / 29

One-way random effects model Maximum likelihood estimates Maximum-likelihood-estimates in the balanced case In the balanced case, n 1 = n 2 = · · · = n k the weights 1 w i ( γ ) = 1 + nγ do not depend on i , and then k � µ = 1 � y i + = y ++ , k i =1 which is the same as the moment estimate. When ( n − 1) SSB > SSE then the maximum likelihood estimate corresponds to an inner point. Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 15 / 29

One-way random effects model Maximum likelihood estimates Maximum-likelihood-estimates in the balanced case, continued Nσ 2 = SSE + SSB 1 + nγ n SSB = SSE + SSB N 1 + nγ k 1 + nγ with the solution SSE σ 2 = � N − k � SSB � γ = 1 � σ 2 − 1 n k � σ 2 b = SSB /k − � σ 2 � n Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 16 / 29

One-way random effects model Estimation of random effects, BLUP-estimation In a mixed effects model, it is not clear what fitted values, and residuals are. Our best prediction for subject i is not given by the mean relationship, µ . It may sometimes be of interest to estimate the random effects. The best linear unbiased predictor (BLUP) in the one-way case is � � µ i = 1 − w i ( γ ) y i + w i ( γ ) µ Thus, the estimate for µ i is a weighted average between the individual bale averages, y i and the overall average � µ with weights (1 − w i ( γ )) and w i ( γ ) , where 1 w i ( γ ) = 1 + n i γ Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 17 / 29

General linear mixed effects models General linear mixed effects models Definition (Linear mixed effects model) The model Y = Xβ + ZU + ǫ with X and Z denoting known matrices, and where ǫ ∼ N ( 0 , Σ ) and U ∼ N ( 0 , Ψ ) are independent is called a mixed general linear model . In the general case may the covariance matrices Σ and Ψ depend on some unknown parameters, ψ , that also need to be estimated. The parameters β are called fixed effects or systematic effects , while the quantities U are called random effects . Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 18 / 29

General linear mixed effects models General linear mixed effects models It follows from the independence of U and ǫ that �� ǫ �� � Σ � 0 D = U 0 Ψ The model may also be interpreted as a hierarchical model U ∼ N ( 0 , Ψ ) Y | U = u ∼ N ( Xβ + Zu , Σ ) Henrik Madsen Poul Thyregod (IMM-DTU) Chapman & Hall January 2011 19 / 29