Interpretation of PROMs: www.kmin-vumc.nl linking measurement error - PowerPoint PPT Presentation

Interpretation of PROMs: www.kmin-vumc.nl linking measurement error to minimal important change Caroline Terwee Knowledgecenter Measurement Instruments Department of Epidemiology and Biostatistics VU University Medical Center

Proposition www.kmin-vumc.nl detectable change is conceptually different from important change

Content www.kmin-vumc.nl • Measurement error (detectable change) – difference from reliability • Important change – anchor-based MIC distribution • Linking measurement error to minimal important change in individual patients • Intermezzo – why distribution-based methods should not be used to define MIC • Alternative ways of interpreting change scores • consider type I error • estimate the probability of belonging to the importantly improved group • Linking measurement error to minimal important change on group level

Measurement error www.kmin-vumc.nl Terminology Minimal Detectable Change Smallest Detectable Change Real change Smallest Real Change Significant change

Measurement error www.kmin-vumc.nl Terminology Minimal Detectable Change Smallest Detectable Change SDC Real change Smallest Real Change Significant change

Measurement error www.kmin-vumc.nl Reliability = The proportion of the total variance in Reliability the measurements which is due to ‘true’ differences Internal between patients Consistency Reliability (test-retest, Inter-rater, Intra-rater) Measurement error = The systematic and random error of a patient’s score that is not attributed to true Measurement error changes in the construct to be measured (test-retest, Inter-rater, Intra-rater) www.cosmin.nl

Measurement error – consistency www.kmin-vumc.nl σ 2 patients = = Reliabilit y ICC σ + σ consistenc y 2 2 patients error

Measurement error – consistency www.kmin-vumc.nl σ 2 patients = = Reliabilit y ICC σ + σ consistenc y 2 2 patients error = σ 2 Standard Error of Measureme nt (SEM ) error consistenc y

Measurement error – consistency www.kmin-vumc.nl σ 2 patients = = Reliabilit y ICC σ + σ consistenc y 2 2 patients error = σ 2 Standard Error of Measureme nt (SEM ) error consistenc y = − SEM SD * 1 ICC consistenc y consistenc y

Measurement error – consistency www.kmin-vumc.nl σ 2 patients = = Reliabilit y ICC σ + σ consistenc y 2 2 patients error = σ 2 Standard Error of Measureme nt (SEM ) error consistenc y = − SEM SD * 1 ICC consistenc y consistenc y ) = Smallest Detectable Change (SDC 1.96 * 2 * SEM consistenc y consistenc y

Measurement error – consistency www.kmin-vumc.nl σ 2 patients = = Reliabilit y ICC σ + σ consistenc y 2 2 patients error = σ 2 Standard Error of Measureme nt (SEM ) error consistenc y = − SEM SD * 1 ICC consistenc y consistenc y ) = Smallest Detectable Change (SDC 1.96 * 2 * SEM consistenc y consistenc y = = SDC 1.96 * SD limits of agreement consistenc y change

Measurement error – agreement www.kmin-vumc.nl σ 2 patients = = Reliabilit y ICC σ + σ + σ agreement 2 2 2 patients measuremen ts error

Measurement error – agreement www.kmin-vumc.nl σ 2 patients = = Reliabilit y ICC σ + σ + σ agreement 2 2 2 patients measuremen ts error = σ + σ 2 2 Standard Error of Measureme nt (SEM ) measuremen ts error agreement

Measurement error – agreement www.kmin-vumc.nl σ 2 patients = = Reliabilit y ICC σ + σ + σ agreement 2 2 2 patients measuremen ts error = σ + σ 2 2 Standard Error of Measureme nt (SEM ) measuremen ts error agreement = − SEM SD * 1 ICC agreement agreement

Measurement error – agreement www.kmin-vumc.nl σ 2 patients = = Reliabilit y ICC σ + σ + σ agreement 2 2 2 patients measuremen ts error = σ + σ 2 2 Standard Error of Measureme nt (SEM ) measuremen ts error agreement = − SEM SD * 1 ICC agreement agreement = SDC 1.96 * 2 * SEM agreement agreement

Measurement error – agreement www.kmin-vumc.nl σ 2 patients = = Reliabilit y ICC σ + σ + σ agreement 2 2 2 patients measuremen ts error = σ + σ 2 2 Standard Error of Measureme nt (SEM ) measuremen ts error agreement = − SEM SD * 1 ICC agreement agreement = SDC 1.96 * 2 * SEM agreement agreement SDC agreement ≠ limits of agreement

Measurement error www.kmin-vumc.nl = SDC 1.96 * 2 * SEM agreement agreement SDC is the smallest change in score that you CAN detect with the instrument, above measurement error in individual patients

Measurement error - example www.kmin-vumc.nl

Measurement error - example www.kmin-vumc.nl Scores 0-100 Measurement error of OES Function = 19 points on a scale from 0-100

Mimimal Important Change (MIC) www.kmin-vumc.nl MIC is the smallest change in score that you WANT to detect with the instrument MIC is determined by an anchor-based method

Mimimal Important Change (MIC) www.kmin-vumc.nl Example anchor: Global rating of change 1. Completely recovered Importantly improved 2. Much improved 3. Slightly improved Not importantly changed 4. No change 5. Slightly worse 6. Much worse

Achor-based MIC distribution www.kmin-vumc.nl ANCHOR ANCHOR Importantly Not importantly improved changed + Change on instrument O _

Anchor-based MIC distribution www.kmin-vumc.nl ANCHOR ANCHOR Importantly Not importantly improved changed + Change on instrument ROC cut-off point O _ MIC = ROC cut-off point

Anchor-based MIC distribution - example www.kmin-vumc.nl

Mimimal Important Change (MIC) - example www.kmin-vumc.nl Improved = ‘slightly better’, ‘much better’, ‘no problems now’ Not improved = ‘no change’, ‘worse’ MIC for OES Function is 5 points

Proposition www.kmin-vumc.nl Smallest Detectable Change (SDC) is conceptually different from Minimal Important Change (MIC) SDC is the smallest change in score that you CAN detect with the instrument, above measurement error MIC is the smallest change in score that you WANT to detect with the instrument

Linking SDC to MIC www.kmin-vumc.nl In individual patients SDC should be smaller than MIC to distinghuish important change from measurement error For may PROMs this is not the case

Linking SDC to MIC - example www.kmin-vumc.nl For OES Function SDC (19) is larger than MIC (5) For OES Pain SDC (8) is smaller than MIC (12.5) Thus with the OES pain we can distinguish important change from measurement erro but with the OES Function this is not possible

Linking SDC to MIC www.kmin-vumc.nl SDC and MIC are two reference points in the scale that can help interpret change scores Example 1 - SDC is smaller than MIC Change statistically significant, but not important Change NOT statistically significant and NOT important Change statistically significant AND important no change maximum change MIC SDC OES Pain - SDC (8) is smaller than MIC (12.5)

Linking SDC to MIC www.kmin-vumc.nl Example 2 - SDC is larger than MIC Change important, but can NOT be distinguished from measurement error Change NOT statistically significant and NOT important Change statistically significant AND important no change maximum change MIC SDC OES Function - SDC (19) is larger than MIC (5)

Intermezzo www.kmin-vumc.nl Why distribution-based methods should not be used to define MIC Distribution-based methods e.g. MIC = 1*SEM or MIC = 0.5*SD

Intermezzo www.kmin-vumc.nl Why distribution-based methods should not be used to define MIC Distribution-based methods e.g. MIC = 1*SEM or MIC = 0.5*SD If MIC is defined as 1*SEM the SDC will always (by definition) be larger than the MIC, because SDC=1.96* √ 2*SEM. This would mean that one can never distinguish important change from measurement error in individual patients

Taking type I error into account www.kmin-vumc.nl

Taking type I error into account www.kmin-vumc.nl = SDC 1.96 * 2 * SEM agreement agreement If we say that a patient who changed as much as the SDC has been ‘really’ changed (statistically significant change), there is a 5% probability (type I error) that in fact this patient has not changed. Patients with change scores smaller than the SDC have a higher probability that they are in fact not changed (larger type 1 error). Thus if the SDC is larger than the MIC, there is a higher type 1 error if we call patients who changed as much as the MIC importantly improved.

Taking type I error into account www.kmin-vumc.nl Change important, but type 1 error may be substantial Change NOT statistically significant and NOT important Type 1 error <5% no change maximum change MIC SDC Type I error

Estimate the probability of belonging to the ‘importantly improved’ group www.kmin-vumc.nl

Estimate the probability of belonging to the ‘importantly improved’ group www.kmin-vumc.nl ANCHOR ANCHOR Importantly Not importantly improved changed + Change on instrument MIC O _