Interpretation of PROMs: www.kmin-vumc.nl linking measurement error - - PowerPoint PPT Presentation

www.kmin-vumc.nl

Interpretation of PROMs: linking measurement error to minimal important change

Caroline Terwee

Knowledgecenter Measurement Instruments Department of Epidemiology and Biostatistics VU University Medical Center

www.kmin-vumc.nl

Proposition detectable change is conceptually different from important change

www.kmin-vumc.nl

Content

• Measurement error (detectable change) –

difference from reliability

• Important change –

anchor-based MIC distribution

• Linking measurement error to minimal important change in individual patients
• Intermezzo –

why distribution-based methods should not be used to define MIC

• Alternative ways of interpreting change scores
• consider type I error
• estimate the probability of belonging to the importantly improved group
• Linking measurement error to minimal important change on group level

www.kmin-vumc.nl

Measurement error Terminology Minimal Detectable Change Smallest Detectable Change Real change Smallest Real Change Significant change

www.kmin-vumc.nl

Measurement error Terminology Minimal Detectable Change Smallest Detectable Change SDC Real change Smallest Real Change Significant change

www.kmin-vumc.nl

Measurement error Reliability = The proportion of the total variance in the measurements which is due to ‘true’ differences between patients Measurement error = The systematic and random error of a patient’s score that is not attributed to true changes in the construct to be measured

www.cosmin.nl

Reliability

Internal Consistency Reliability (test-retest, Inter-rater, Intra-rater) Measurement error (test-retest, Inter-rater, Intra-rater)

www.kmin-vumc.nl

Measurement error – consistency error patients patients ICC y Reliabilit

y consistenc 2 2 2

σ σ σ + = =

www.kmin-vumc.nl

Measurement error – consistency error patients patients ICC y Reliabilit

y consistenc 2 2 2

σ σ σ + = =

error ) (SEM nt Measureme

• f

Error Standard

y consistenc 2

σ =

www.kmin-vumc.nl

Measurement error – consistency error patients patients ICC y Reliabilit

y consistenc 2 2 2

σ σ σ + = =

ICC SD SEM

y consistenc y consistenc

− = 1 * error ) (SEM nt Measureme

• f

Error Standard

y consistenc 2

σ =

www.kmin-vumc.nl

Measurement error – consistency error patients patients ICC y Reliabilit

y consistenc 2 2 2

σ σ σ + = =

y consistenc y consistenc

SEM 2 * 1.96 (SDC Change Detectable Smallest * ) =

ICC SD SEM

y consistenc y consistenc

− = 1 * error ) (SEM nt Measureme

• f

Error Standard

y consistenc 2

σ =

www.kmin-vumc.nl

Measurement error – consistency error patients patients ICC y Reliabilit

y consistenc 2 2 2

σ σ σ + = =

error ) (SEM nt Measureme

• f

Error Standard

y consistenc 2

σ =

agreement

• f

limits SD * 1.96 SDC

change y consistenc

= =

y consistenc y consistenc

SEM 2 * 1.96 (SDC Change Detectable Smallest * ) =

ICC SD SEM

y consistenc y consistenc

− = 1 *

www.kmin-vumc.nl

Measurement error – agreement error ts measuremen patients patients ICC y Reliabilit

agreement 2 2 2 2

σ σ σ σ + + = =

www.kmin-vumc.nl

Measurement error – agreement

error ts measuremen ) (SEM nt Measureme

• f

Error Standard

agreement 2 2

σ σ + =

error ts measuremen patients patients ICC y Reliabilit

agreement 2 2 2 2

σ σ σ σ + + = =

www.kmin-vumc.nl

Measurement error – agreement

error ts measuremen ) (SEM nt Measureme

• f

Error Standard

agreement 2 2

σ σ + =

error ts measuremen patients patients ICC y Reliabilit

agreement 2 2 2 2

σ σ σ σ + + = =

ICC SD SEM

agreement agreement

− = 1 *

www.kmin-vumc.nl

Measurement error – agreement

agreement agreement

SEM 2 * 1.96 SDC * =

error ts measuremen ) (SEM nt Measureme

• f

Error Standard

agreement 2 2

σ σ + =

error ts measuremen patients patients ICC y Reliabilit

agreement 2 2 2 2

σ σ σ σ + + = =

ICC SD SEM

agreement agreement

− = 1 *

www.kmin-vumc.nl

Measurement error – agreement

agreement agreement

SEM 2 * 1.96 SDC * =

error ts measuremen ) (SEM nt Measureme

• f

Error Standard

agreement 2 2

σ σ + =

error ts measuremen patients patients ICC y Reliabilit

agreement 2 2 2 2

σ σ σ σ + + = =

ICC SD SEM

agreement agreement

− = 1 *

agreement

• f

limits SDCagreement ≠

www.kmin-vumc.nl

Measurement error

agreement agreement

SEM 2 * 1.96 SDC * =

SDC is the smallest change in score that you CAN detect with the instrument, above measurement error in individual patients

www.kmin-vumc.nl

Measurement error - example

www.kmin-vumc.nl

Measurement error - example

Scores 0-100

Measurement error of OES Function = 19 points on a scale from 0-100

www.kmin-vumc.nl

Mimimal Important Change (MIC) MIC is the smallest change in score that you WANT to detect with the instrument MIC is determined by an anchor-based method

www.kmin-vumc.nl

Mimimal Important Change (MIC)

Example anchor: Global rating of change 1. Completely recovered 2. Much improved 3. Slightly improved 4. No change 5. Slightly worse 6. Much worse Importantly improved Not importantly changed

www.kmin-vumc.nl

Achor-based MIC distribution

+ _

Importantly improved

ANCHOR ANCHOR

Not importantly changed

O

Change on instrument

www.kmin-vumc.nl

Anchor-based MIC distribution

+ _

Importantly improved

ANCHOR ANCHOR

Not importantly changed

O

Change on instrument ROC cut-off point MIC = ROC cut-off point

www.kmin-vumc.nl

Anchor-based MIC distribution - example

www.kmin-vumc.nl

Mimimal Important Change (MIC) - example

Improved = ‘slightly better’, ‘much better’, ‘no problems now’ Not improved = ‘no change’, ‘worse’

MIC for OES Function is 5 points

www.kmin-vumc.nl

Proposition Smallest Detectable Change (SDC) is conceptually different from Minimal Important Change (MIC)

SDC is the smallest change in score that you CAN detect with the instrument, above measurement error MIC is the smallest change in score that you WANT to detect with the instrument

www.kmin-vumc.nl

Linking SDC to MIC In individual patients SDC should be smaller than MIC to distinghuish important change from measurement error For may PROMs this is not the case

www.kmin-vumc.nl

Linking SDC to MIC - example For OES Function SDC (19) is larger than MIC (5) For OES Pain SDC (8) is smaller than MIC (12.5) Thus with the OES pain we can distinguish important change from measurement erro but with the OES Function this is not possible

www.kmin-vumc.nl

Linking SDC to MIC SDC and MIC are two reference points in the scale that can help interpret change scores Example 1 - SDC is smaller than MIC

Change NOT statistically significant and NOT important Change statistically significant AND important Change statistically significant, but not important SDC MIC no change maximum change

OES Pain - SDC (8) is smaller than MIC (12.5)

www.kmin-vumc.nl

Change NOT statistically significant and NOT important Change statistically significant AND important Change important, but can NOT be distinguished from measurement error SDC MIC no change maximum change

Linking SDC to MIC Example 2 - SDC is larger than MIC OES Function - SDC (19) is larger than MIC (5)

www.kmin-vumc.nl

Intermezzo Why distribution-based methods should not be used to define MIC Distribution-based methods e.g. MIC = 1*SEM or MIC = 0.5*SD

www.kmin-vumc.nl

Intermezzo Why distribution-based methods should not be used to define MIC Distribution-based methods e.g. MIC = 1*SEM or MIC = 0.5*SD If MIC is defined as 1*SEM the SDC will always (by definition) be larger than the MIC, because SDC=1.96*√2*SEM. This would mean that one can never distinguish important change from measurement error in individual patients

www.kmin-vumc.nl

Taking type I error into account

www.kmin-vumc.nl

Taking type I error into account

agreement agreement

SEM 2 * 1.96 SDC * =

If we say that a patient who changed as much as the SDC has been ‘really’ changed (statistically significant change), there is a 5% probability (type I error) that in fact this patient has not changed. Patients with change scores smaller than the SDC have a higher probability that they are in fact not changed (larger type 1 error). Thus if the SDC is larger than the MIC, there is a higher type 1 error if we call patients who changed as much as the MIC importantly improved.

www.kmin-vumc.nl

Change NOT statistically significant and NOT important Type 1 error <5% Change important, but type 1 error may be substantial SDC MIC no change maximum change

Taking type I error into account Type I error

www.kmin-vumc.nl

Estimate the probability of belonging to the ‘importantly improved’ group

www.kmin-vumc.nl

Estimate the probability of belonging to the ‘importantly improved’ group

+ _

Importantly improved

ANCHOR ANCHOR

Not importantly changed

O

Change on instrument MIC

www.kmin-vumc.nl

Estimate the probability of belonging to the ‘importantly improved’ group Logistic regression analysis Dependent variable = group (0=not importantly changed,1= importantly improved) Independent variable = change score on the instrument

) (

1 1

X

e y probabilit predicted

β α + −

+ =

www.kmin-vumc.nl

Example Hypothetical health questionnaire, score 0-100 (higer score = better health) Sample n=20.000 Anchor: 50% = importantly improved 50% = not importantly changed Prior probability of belonging to the ‘importantly improved’ group = 50%

www.kmin-vumc.nl

B S.E. Wald df Sig. Exp(B) 95,0% C.I.for EXP(B) Lower Upper Change score ,050 ,001 4005,047 1 ,000 1,052 1,050 1,053 Constant

• ,966

,022 1940,184 1 ,000 ,381

Example Logistic regression analysis α=-0.966 and β=0.050

) 050 . 966 . (

1 1

X

e y probabilit predicted

+ − −

+ =

www.kmin-vumc.nl

Example

0.28 e p = + =

+ − − ) * 050 . 966 . (

1 1

Patient 1 – change score = 0 Patient 3 – change score = 50

0.82 e p = + =

+ − − ) 50 * 050 . 966 . (

1 1

Patient 2 – change score = 25

0.43 e p = + =

+ − − ) 25 * 050 . 966 . (

1 1

www.kmin-vumc.nl

Example Predicted probabilities of belonging to the ‘importantly improved’ group for each possible change score on the questionnaire

Change score on instrument Predicted probability

www.kmin-vumc.nl

Finally linking SDC to MIC – on group level In research we use mean change scores of groups Measurement error is reduced because of repeated measurements

n SDC SDC

individual group =

MIC on group level = smaller ???

www.kmin-vumc.nl

Summary Smallest Detectable Change (SDC) is conceptually different from Minimal Important Change (MIC) SDC is the smallest change in score that you CAN detect with the instrument, above measurement error preferably based on SEMagreement MIC is the smallest change in score that you WANT to detect with the instrument Should be based on anchor-based methods The SDC and the MIC are two important reference points on a scale that can help interpret change scores in individual patients As an alternative, estimating the probability of belonging to the importantly improved group may be useful

www.kmin-vumc.nl

Thank you cb.terwee@vumc.nl

Terwee CB, Roorda LD, Knol DL, de Boer MR, de Vet HCW. Linking measurement error to minimal important change of patient-reported outcomes. J Clin Epidemiol 2009;62:1062-1067. de Vet HCW, Ostelo RWJG, Terwee CB, van der Roer N, Knol DL, Beckerman H, Boers M, Bouter LM. Minimally import ant change determined by a visual method integrating an anchor- based and a distribution-based approach. Qual Life Res 2007;16:131-142. de Vet HCW, Terwee CB, Knol DL, Bouter LM. When to use agreement versus reliability measures. J Clin Epidemiol 2006;59:1033-1039.