Generalized Ruuds Theorem Mariusz Kubkowski a , Jan Mielniczuk a , b - PowerPoint PPT Presentation

Generalized Ruud’s Theorem Mariusz Kubkowski a , Jan Mielniczuk a , b a Warsaw University of Technology Faculty of Mathematics and Information Science, b Institute of Computer Science Polish Academy of Sciences Będlewo, December 1st 2016 M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 1 / 29

Introduction p , k ∈ N , k ≤ p , M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 2 / 29

Introduction p , k ∈ N , k ≤ p , ( X , Y ) ∈ R p × { 0 , 1 } - random vector, X ∼ P X , M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 2 / 29

Introduction p , k ∈ N , k ≤ p , ( X , Y ) ∈ R p × { 0 , 1 } - random vector, X ∼ P X , q : R k → [ 0 , 1 ] - unknown response function, M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 2 / 29

Introduction p , k ∈ N , k ≤ p , ( X , Y ) ∈ R p × { 0 , 1 } - random vector, X ∼ P X , q : R k → [ 0 , 1 ] - unknown response function, β 1 , . . . , β k ∈ R p , - true coefficients, � � ∈ R p × k . B = β 1 . . . β k M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 2 / 29

Introduction p , k ∈ N , k ≤ p , ( X , Y ) ∈ R p × { 0 , 1 } - random vector, X ∼ P X , q : R k → [ 0 , 1 ] - unknown response function, β 1 , . . . , β k ∈ R p , - true coefficients, � � ∈ R p × k . B = β 1 . . . β k Conditional distribution of Y | X : P ( Y = 1 | X ) = q ( β T 1 X , . . . , β T k X ) =: q ( B T X ) . M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 2 / 29

Introduction β 01 , . . . , β 0 k ∈ R , q ( B T X ) = ˜ q ( β 01 + β T 1 X , . . . , β 0 k + β T k X ) . M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 3 / 29

Introduction β 01 , . . . , β 0 k ∈ R , q ( B T X ) = ˜ q ( β 01 + β T 1 X , . . . , β 0 k + β T k X ) . Special case ( k = 1 , B = β 1 ): P ( Y = 1 | X ) = q ( β T 1 X ) . M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 3 / 29

Misspecification problem We fit the model: P ( Y = 1 | X ) = q L ( β ∗ 0 + β ∗ T X ) - model M 0 (with intercept) M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 4 / 29

Misspecification problem We fit the model: P ( Y = 1 | X ) = q L ( β ∗ 0 + β ∗ T X ) - model M 0 (with intercept) or P ( Y = 1 | X ) = q L ( β ∗ T X ) - model M (without intercept) , M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 4 / 29

Misspecification problem We fit the model: P ( Y = 1 | X ) = q L ( β ∗ 0 + β ∗ T X ) - model M 0 (with intercept) or P ( Y = 1 | X ) = q L ( β ∗ T X ) - model M (without intercept) , where: 1 1 + e − x , x ∈ R . q L ( x ) = M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 4 / 29

Misspecification problem Log-likelihood function ( b 0 ∈ R , b ∈ R p ): � � �� l ( b 0 , b , X , Y ) = Y ( b 0 + b T X ) − log b 0 + b T X 1 + exp . M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 5 / 29

Misspecification problem Log-likelihood function ( b 0 ∈ R , b ∈ R p ): � � �� l ( b 0 , b , X , Y ) = Y ( b 0 + b T X ) − log b 0 + b T X 1 + exp . We want to find coefficients (in the model M 0 ): ( β ∗ 0 , β ∗ ) = ( b 0 , b ) ∈ R × R p E ( X , Y ) l ( b 0 , b , X , Y ) arg max or (in the model M ): β ∗ = arg max E ( X , Y ) l ( 0 , b , X , Y ) . b ∈ R p M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 5 / 29

Misspecification problem Theorem (Li, Duan, 1989) If q ( B T X ) ∈ ( 0 , 1 ) a.s. and E || X || < ∞ , then there exist solutions for models M 0 and M . M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 6 / 29

Misspecification problem Theorem (Li, Duan, 1989) If q ( B T X ) ∈ ( 0 , 1 ) a.s. and E || X || < ∞ , then there exist solutions for models M 0 and M . Theorem (Li, Duan, 1989) If q ( B T X ) ∈ ( 0 , 1 ) a.s., E || X || 2 < ∞ , and E XX T > 0 , then the solutions for models M 0 and M are unique. M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 6 / 29

Misspecification problem Theorem (Li, Duan, 1989) If q ( B T X ) ∈ ( 0 , 1 ) a.s. and E || X || < ∞ , then there exist solutions for models M 0 and M . Theorem (Li, Duan, 1989) If q ( B T X ) ∈ ( 0 , 1 ) a.s., E || X || 2 < ∞ , and E XX T > 0 , then the solutions for models M 0 and M are unique. Normal equations (model M 0 ): X T � T ( q ( B T X ) − q L ( β ∗ E ( X , Y ) D ( b 0 , b ) l ( β ∗ 0 , β ∗ , X , Y ) = E 0 + β ∗ T X )) = 0 , � 1 equivalently: X T � T = E q ( B T X ) X T � T . E q L ( β ∗ 0 + β ∗ T X ) � � 1 1 M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 6 / 29

Ruud’s theorem ( k = 1) Ruud’s condition for B and k = 1 (Ruud, 1983) ∀ z ∈ R ∃ u 0 , u ∈ R p : E ( X | β T 1 X = z ) = u 0 + uz Remark: Ruud’s condition is satisfied, when X has elliptically contoured distribution, in particular normal distribution. M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 7 / 29

Ruud’s theorem ( k = 1) Ruud’s condition for B and k = 1 (Ruud, 1983) ∀ z ∈ R ∃ u 0 , u ∈ R p : E ( X | β T 1 X = z ) = u 0 + uz Remark: Ruud’s condition is satisfied, when X has elliptically contoured distribution, in particular normal distribution. Theorem (Ruud, 1983) If k = 1 , q ( β T 1 X ) ∈ ( 0 , 1 ) a.s., X satisfies Ruud’s condition for B and k = 1 , E || X || 2 < ∞ , and E XX T > 0 , then in the models M and M 0 there exists η ∈ R such that: β ∗ = ηβ 1 . M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 7 / 29

Generalized Ruud’s theorem Ruud’s condition for B (Li, 1991) k ∀ z ∈ R k ∃ u 0 ∈ R p , U ∈ R p × k : E ( X | B T X = z ) = u 0 + U z = u 0 + � U ( i ) z i i = 1 M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 8 / 29

Generalized Ruud’s theorem Ruud’s condition for B (Li, 1991) k ∀ z ∈ R k ∃ u 0 ∈ R p , U ∈ R p × k : E ( X | B T X = z ) = u 0 + U z = u 0 + � U ( i ) z i i = 1 Theorem If q ( B T X ) ∈ ( 0 , 1 ) a.s., X satisfies Ruud’s condition for B , E || X || 2 < ∞ , and E XX T > 0 , then the coefficients of the logistic models M and M 0 satisfy the following condition: k ∃ η ∈ R k : β ∗ = B η = � η i β i . i = 1 M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 8 / 29

Form of η Theorem (representation of η ) If the assumptions of the generalized Ruud’s theorem are satisfied, rank B = k , and additionally X satisfies Ruud’s condition for β ∗ , then η from the generalized Ruud’s theorem satisfies the following equation: a β ∗ · η = a B , where a G = ( Var ( G T X )) − 1 Cov ( G T X , Y ) for G ∈ R p × l ( l ∈ N ) - full column rank matrix. Moreover, if Cov ( B T X , Y ) � = 0 , then a β ∗ � = 0 . M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 9 / 29

Form of η Theorem (representation of η ) If the assumptions of the generalized Ruud’s theorem are satisfied, rank B = k , and additionally X satisfies Ruud’s condition for β ∗ , then η from the generalized Ruud’s theorem satisfies the following equation: a β ∗ · η = a B , where a G = ( Var ( G T X )) − 1 Cov ( G T X , Y ) for G ∈ R p × l ( l ∈ N ) - full column rank matrix. Moreover, if Cov ( B T X , Y ) � = 0 , then a β ∗ � = 0 . Remark: If X ∼ N p ( µ, Σ) , Σ > 0 , q - differentiable and E | Dq ( B T X ) | < ∞ , then from the Stein’s lemma: a B = E Dq ( B T X ) , a β ∗ = E q ′ L ( β ∗ T X ) . M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 9 / 29

Algorithm for finding η X ∼ N ( µ, Σ) , Σ > 0 , rank B = k . Let (model M 0 ): U = B T X , ˜ η = [ β ∗ U = [ 1 U T ] T , ˜ 0 η T ] T . M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 10 / 29

Algorithm for finding η X ∼ N ( µ, Σ) , Σ > 0 , rank B = k . Let (model M 0 ): U = B T X , ˜ η = [ β ∗ U = [ 1 U T ] T , ˜ 0 η T ] T . Consider the function: η T ˜ U )˜ U − E q ( U )˜ F (˜ η ) = E q L (˜ U , Newton-Raphson iterations ( ˜ η 0 - fixed): η n )) − 1 F (˜ η n + 1 = ˜ ˜ η n − ( DF (˜ η n ) . Problem: how to compute expected values? M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 10 / 29

Algorithm for finding η X ∼ N ( µ, Σ) , Σ > 0 , rank B = k . Let (model M 0 ): U = B T X , ˜ η = [ β ∗ U = [ 1 U T ] T , ˜ 0 η T ] T . Consider the function: η T ˜ U )˜ U − E q ( U )˜ F (˜ η ) = E q L (˜ U , Newton-Raphson iterations ( ˜ η 0 - fixed): η n )) − 1 F (˜ η n + 1 = ˜ ˜ η n − ( DF (˜ η n ) . Problem: how to compute expected values? Gauss - Hermite quadratures (from R package fastGHQuad ). M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 10 / 29

Additive misspecification - special case Let: k � q ( B T X ) = λ i q i ( β T i X ) , i = 1 k � where for each i = 1 , . . . , k : q i : R → ( 0 , 1 ) , λ i ≥ 0 , λ i = 1 . i = 1 M. Kubkowski, J. Mielniczuk Generalized Ruud’s Theorem 2016-12-01 11 / 29