How black-box use of imputation can cause bias Nicole Erler Erasmus - - PowerPoint PPT Presentation

▶
how black box use of imputation can cause bias
SMART_READER_LITE
LIVE PREVIEW

How black-box use of imputation can cause bias Nicole Erler Erasmus - - PowerPoint PPT Presentation

Jul 20, 2023 579 likes •1.03k views

How black-box use of imputation can cause bias Nicole Erler Erasmus Medical Center, Rotterdam Nicole Erler, FGME 2019, Kiel 1 Handling Missing Values is Easy! Functions automatically exclude missing values: ## [...] ## Residual standard

slide-1
SLIDE 1

How black-box use of imputation can cause bias

Nicole Erler

Erasmus Medical Center, Rotterdam

Nicole Erler, FGME 2019, Kiel 1

slide-2
SLIDE 2

Handling Missing Values is Easy!

Functions automatically exclude missing values: ## [...] ## Residual standard error: 2.305 on 69 degrees of freedom ## (25 observations deleted due to missingness) ## Multiple R-squared: 0.09255, Adjusted R-squared: 0.02679 ## F-statistic: 1.407 on 5 and 69 DF, p-value: 0.2325

Nicole Erler, FGME 2019, Kiel 2

slide-3
SLIDE 3

Handling Missing Values is Easy!

Functions automatically exclude missing values: ## [...] ## Residual standard error: 2.305 on 69 degrees of freedom ## (25 observations deleted due to missingness) ## Multiple R-squared: 0.09255, Adjusted R-squared: 0.02679 ## F-statistic: 1.407 on 5 and 69 DF, p-value: 0.2325 Imputation is super easy: library("mice") imp <- mice(mydata) However ...

Nicole Erler, FGME 2019, Kiel 2

slide-4
SLIDE 4

Handling Missing Values Correctly is Not So Easy!

(Imputation) methods makes certain assumptions, e.g.: missingness is M(C)AR

Nicole Erler, FGME 2019, Kiel 3

slide-5
SLIDE 5

Handling Missing Values Correctly is Not So Easy!

(Imputation) methods makes certain assumptions, e.g.: missingness is M(C)AR the incomplete variable has a certain conditional distribution (e.g. normal)

Nicole Erler, FGME 2019, Kiel 3

slide-6
SLIDE 6

Handling Missing Values Correctly is Not So Easy!

(Imputation) methods makes certain assumptions, e.g.: missingness is M(C)AR the incomplete variable has a certain conditional distribution (e.g. normal) all associations are linear

no interactions no non-linear effects no transformations

Nicole Erler, FGME 2019, Kiel 3

slide-7
SLIDE 7

Handling Missing Values Correctly is Not So Easy!

(Imputation) methods makes certain assumptions, e.g.: missingness is M(C)AR the incomplete variable has a certain conditional distribution (e.g. normal) all associations are linear

no interactions no non-linear effects no transformations

compatibility of the imputation models congeniality (compatibility between analysis and imputation models)

Nicole Erler, FGME 2019, Kiel 3

slide-8
SLIDE 8

Handling Missing Values Correctly is Not So Easy!

(Imputation) methods makes certain assumptions, e.g.: missingness is M(C)AR the incomplete variable has a certain conditional distribution (e.g. normal) all associations are linear

no interactions no non-linear effects no transformations

compatibility of the imputation models congeniality (compatibility between analysis and imputation models)

violation ➡ bias

Nicole Erler, FGME 2019, Kiel 3

slide-9
SLIDE 9

Literature: mis-specification in Multiple Imputation

Several authors have investigated robustness to mis-specification (of distribution)

in MI using FCS / MICE in joint model MI

and/or proposed to use

Tukey’s gh distribution Fleishman polynomials GAMs (in FCS) Doubly-robust weighted estimating equations (instead of MI)

Nicole Erler, FGME 2019, Kiel 4

slide-10
SLIDE 10

Fully Bayesian Analysis & Imputation

Joint distribution p(y | X, b, θ)

  • analysis

model p(X | θ)

  • imputation

part p(b | θ)

  • random

effects p(θ)

  • priors

Nicole Erler, FGME 2019, Kiel 5

slide-11
SLIDE 11

Fully Bayesian Analysis & Imputation

Joint distribution p(y | X, b, θ)

  • analysis

model p(X | θ)

  • imputation

part p(b | θ)

  • random

effects p(θ)

  • priors

Imputation part p(

X

  • x1, . . . , xp, Xcompl. | θ)

= p(x1 | Xcompl., θ) p(x2 | Xcompl., x1, θ) p(x3 | Xcompl., x1, x2, θ) . . .

Nicole Erler, FGME 2019, Kiel 5

slide-12
SLIDE 12

Fully Bayesian Analysis & Imputation

Joint distribution p(y | X, b, θ)

  • analysis

model p(X | θ)

  • imputation

part p(b | θ)

  • random

effects p(θ)

  • priors

Imputation part p(

X

  • x1, . . . , xp, Xcompl. | θ)

= p(x1 | Xcompl., θ) p(x2 | Xcompl., x1, θ) p(x3 | Xcompl., x1, x2, θ) . . . Software Implemented in the R package JointAI

Nicole Erler, FGME 2019, Kiel 5

slide-13
SLIDE 13

MICE vs JointAI

Imputation in MICE p(x1 | y, Xcompl., x2, x3, x4, . . . , θ) p(x2 | y, Xcompl., x1, x3, x4, . . . , θ) p(x3 | y, Xcompl., x1, x2, x4, . . . , θ) . . . Imputation in JointAI p(y | Xcompl., x1, x2, x3, . . . , θ) p(x1 | Xcompl., θ) p(x2 | Xcompl., x1, θ) p(x3 | Xcompl., x1, x2, θ) . . .

Nicole Erler, FGME 2019, Kiel 6

slide-14
SLIDE 14

MICE vs JointAI

Imputation in MICE p(x1 | y, Xcompl., x2, x3, x4, . . . , θ) p(x2 | y, Xcompl., x1, x3, x4, . . . , θ) p(x3 | y, Xcompl., x1, x2, x4, . . . , θ) . . . Imputation in JointAI p(y | Xcompl., x1, x2, x3, . . . , θ) p(x1 | Xcompl., θ) p(x2 | Xcompl., x1, θ) p(x3 | Xcompl., x1, x2, θ) . . . No issues with complex outcomes, e.g.:

multi-level survival

congeniality compatibility

Nicole Erler, FGME 2019, Kiel 6

slide-15
SLIDE 15

MICE vs JointAI

Imputation in MICE p(x1 | y, Xcompl., x2, x3, x4, . . . , θ) p(x2 | y, Xcompl., x1, x3, x4, . . . , θ) p(x3 | y, Xcompl., x1, x2, x4, . . . , θ) . . .

➡

Imputation in JointAI p(y | Xcompl., x1, x2, x3, . . . , θ) p(x1 | Xcompl., θ) p(x2 | Xcompl., x1, θ) p(x3 | Xcompl., x1, x2, θ) . . .

➡

Potential mis-specification of association structure conditional distribution M(C)AR

Nicole Erler, FGME 2019, Kiel 7

slide-16
SLIDE 16

Simulation Study: Quadratic Effect

Analysis model: y ∼ β0 + β1x1 + β2x2 + . . . Quadratic association between covariates: x1 ∼ α0 + α1x2 +❍❍

❍

α2x2

2 + . . . −2 2 −2 2 −2 2

−10 10 20 30

x2(complete covariate) x1(incomplete covariate)

Nicole Erler, FGME 2019, Kiel 8

slide-17
SLIDE 17

Simulation Study: Quadratic Effect

Analysis model: y ∼ β0 + β1x1 + β2x2 + . . . Quadratic association between covariates: x1 ∼ α0 + α1x2 +❍❍

❍

α2x2

2 + . . . −2 2 −2 2 −2 2

−10 10 20 30

x2(complete covariate) x1(incomplete covariate) mice JointAI mice JointAI mice JointAI

0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

relative bias

10% NA 30% NA 50% NA Nicole Erler, FGME 2019, Kiel 8

slide-18
SLIDE 18

Simulation Study: Logarithmic Effect

Analysis model: y ∼ β0 + β1x1 + β2x2 + . . . Log-association between covariates: x1 ∼ α0 + α1❩

❩

log(x2) + . . .

0.4 0.8 1.2 0.4 0.8 1.2 0.4 0.8 1.2

−12 −8 −4 4

x2(complete covariate) x1(incomplete covariate) mice JointAI mice JointAI mice JointAI

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8

relative bias

10% NA 30% NA 50% NA Nicole Erler, FGME 2019, Kiel 9

slide-19
SLIDE 19

Simulation Study: Gamma distribution

Analysis model: y ∼ β0 + β1x1 + β2x2 + . . . Gamma-distributed covariate: x1 | x2, x3, . . . ∼ Ga()

2 4 6 2 4 6 2 4 6

0.0 0.5 1.0 1.5 2.0

x1(incomplete covariate) conditional distribution mice JointAI mice JointAI mice JointAI

0.8 1.0 1.2 1.4 1.6 1.8 2.0

relative bias

10% NA 30% NA 50% NA Nicole Erler, FGME 2019, Kiel 10

slide-20
SLIDE 20

Flexible Bayesian Models

We need more flexible imputation models! Ideally: models that fit (almost) any distribution / association structure.

Nicole Erler, FGME 2019, Kiel 11

slide-21
SLIDE 21

Flexible Bayesian Models

We need more flexible imputation models! Ideally: models that fit (almost) any distribution / association structure. Ideas: flexible association structure: penalized splines flexible residual distribution: mixture of Polya-Trees

Nicole Erler, FGME 2019, Kiel 11

slide-22
SLIDE 22

Bayesian P-Splines

Instead of β1x2 we use

d

  • ℓ=1

βℓBℓ(x2):

  • ●
  • ●
  • x2

x1

Nicole Erler, FGME 2019, Kiel 12

slide-23
SLIDE 23

Bayesian P-Splines

Instead of β1x2 we use

d

  • ℓ=1

βℓBℓ(x2):

  • ●
  • ●
  • x2

x1

Nicole Erler, FGME 2019, Kiel 12

slide-24
SLIDE 24

Bayesian P-Splines

Instead of β1x2 we use

d

  • ℓ=1

βℓBℓ(x2):

  • ●
  • ●
  • x2

x1

Nicole Erler, FGME 2019, Kiel 12

slide-25
SLIDE 25

Simulation: Bayesian P-Splines

Analysis model: y ∼ β0 + β1x1 + β2x2 + . . . Quadratic association between covariates: x1 ∼ α0 + α1x2 +❍❍

❍

α2x2

2 + . . . −2 2 −2 2 −2 2

−10 10 20 30

x2(complete covariate) x1(incomplete covariate)

Nicole Erler, FGME 2019, Kiel 13

slide-26
SLIDE 26

Simulation: Bayesian P-Splines

Analysis model: y ∼ β0 + β1x1 + β2x2 + . . . Quadratic association between covariates: x1 ∼ α0 + α1x2 +❍❍

❍

α2x2

2 + . . . −2 2 −2 2 −2 2

−10 10 20 30

x2(complete covariate) x1(incomplete covariate) JointAI default JointAI p−spline JointAI default JointAI p−spline JointAI default JointAI p−spline

1.00 1.25 1.50

relative bias

Nicole Erler, FGME 2019, Kiel 13

slide-27
SLIDE 27

Simulation: Bayesian P-Splines

Potential Issue: covariate incomplete covariate

missing

  • bserved

Nicole Erler, FGME 2019, Kiel 14

slide-28
SLIDE 28

Simulation: Bayesian P-Splines

Potential Issue: covariate incomplete covariate

missing

  • bserved

Nicole Erler, FGME 2019, Kiel 14

slide-29
SLIDE 29

Mixture of Polya Trees

0.0 0.5 1.0 1.5

density Beta(a0, a0)

Nicole Erler, FGME 2019, Kiel 15

slide-30
SLIDE 30

Mixture of Polya Trees

0.0 0.5 1.0 1.5

density Beta(a0, a0) Beta(a1, a1) Beta(a1, a1)

Nicole Erler, FGME 2019, Kiel 15

slide-31
SLIDE 31

Mixture of Polya Trees

0.0 0.5 1.0 1.5

density Beta(a0, a0) Beta(a1, a1) Beta(a1, a1) Beta(a2, a2) Beta(a2, a2) Beta(a2, a2) Beta(a2, a2)

Nicole Erler, FGME 2019, Kiel 15

slide-32
SLIDE 32

Mixture of Polya Trees

0.0 0.5 1.0 1.5

density Beta(a0, a0) Beta(a1, a1) Beta(a1, a1) Beta(a2, a2) Beta(a2, a2) Beta(a2, a2) Beta(a2, a2) Beta(a3, a3) Beta(a3, a3) Beta(a3, a3) Beta(a3, a3)

Nicole Erler, FGME 2019, Kiel 15

slide-33
SLIDE 33

Mixture of Polya Trees

0.0 0.5 1.0 1.5

density Beta(a0, a0) Beta(a1, a1) Beta(a1, a1) Beta(a2, a2) Beta(a2, a2) Beta(a2, a2) Beta(a2, a2) Beta(a3, a3) Beta(a3, a3) Beta(a3, a3) Beta(a3, a3)

Nicole Erler, FGME 2019, Kiel 15

slide-34
SLIDE 34

Mixture of Polya Trees

0.0 0.5 1.0 1.5

density Beta(a0, a0) Beta(a1, a1) Beta(a1, a1) Beta(a2, a2) Beta(a2, a2) Beta(a2, a2) Beta(a2, a2) Beta(a3, a3) Beta(a3, a3) Beta(a3, a3) Beta(a3, a3)

Nicole Erler, FGME 2019, Kiel 15

slide-35
SLIDE 35

Mixture of Polya Trees

0.0 0.5 1.0 1.5

density Beta(a0, a0) Beta(a1, a1) Beta(a1, a1) Beta(a2, a2) Beta(a2, a2) Beta(a2, a2) Beta(a2, a2) Beta(a3, a3) Beta(a3, a3) Beta(a3, a3) Beta(a3, a3)

Nicole Erler, FGME 2019, Kiel 15

slide-36
SLIDE 36

Practical Issues & Ideas

Potential / probable issues for practice: flexible fit needs observed data everywhere computational time

Nicole Erler, FGME 2019, Kiel 16

slide-37
SLIDE 37

Practical Issues & Ideas

Potential / probable issues for practice: flexible fit needs observed data everywhere computational time Ideas: check first if simple model fits, e.g. posterior predictive checks

χ2 type of tests Kolmogorov-Smirnoff test? discordance tests?

Nicole Erler, FGME 2019, Kiel 16

slide-38
SLIDE 38

Practical Issues & Ideas

Potential / probable issues for practice: flexible fit needs observed data everywhere computational time Ideas: check first if simple model fits, e.g. posterior predictive checks

χ2 type of tests Kolmogorov-Smirnoff test? discordance tests?

feasibility checks before running the complex model

Nicole Erler, FGME 2019, Kiel 16

slide-39
SLIDE 39

Take home message

assumptions of imputation models can easily be violated ➡ bias more flexible imputation models are needed semi- / non-parametric (Bayesian) methods can offer a solution more flexibility ➡ more complexity ➡ need for guidance tools

Nicole Erler, FGME 2019, Kiel 17

slide-40
SLIDE 40

Thank you for your attention.

  • n.erler@erasmusmc.nl
  • N_Erler
  • NErler
  • www.nerler.com
slide-41
SLIDE 41

Simulation Study: Beta Distribution

Analysis model: y ∼ β0 + β1x1 + β2 + . . . Beta-distributed covariate: x1 | x2, x3, . . . ∼ Be()

0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00

1 2 3 4

x1(incomplete covariate) conditional distribution mice JointAI mice JointAI mice JointAI

0.6 0.8 1.0 1.2 1.4

relative bias

Nicole Erler, FGME 2019, Kiel 19

slide-42
SLIDE 42

Simulation Study: Reverting the sequence

Analysis model: y ∼ β0 + β1x1 + β2x2 + . . . Exclusion of important predictor: x1 ∼ α0 + α1x2 +❳❳

❳

α2x3 + . . .

−5 5 −5 5 −5 5

0.0 0.2 0.4 0.6

x1(incomplete covariate) conditional distribution mice JointAI mice JointAI mice JointAI

0.6 0.8 1.0 1.2 1.4 1.6

relative bias

10% NA 30% NA 50% NA Nicole Erler, FGME 2019, Kiel 20

slide-43
SLIDE 43

Simulation Study: Ignoring an Interaction

Analysis model: y ∼ β0 + β1x1 + β2x2 + . . . Interaction between covariates: x1 ∼ α0 + α1x2 + α2x3 +❳❳❳

❳

α3x2x3 + . . .

−2 −1 1 2 −2 −1 1 2 −2 −1 1 2

−5.0 −2.5 0.0 2.5 5.0

x2(complete covariate) x1(incomplete covariate) mice JointAI mice JointAI mice JointAI

0.6 0.8 1.0 1.2 1.4 1.6 1.8

relative bias

10% NA 30% NA 50% NA Nicole Erler, FGME 2019, Kiel 21