A Rational Approach to Outcome Measurement Duncan Raistrick Leeds - - PowerPoint PPT Presentation

A Rational Approach to Outcome Measurement

Duncan Raistrick Leeds Addiction Unit

Leeds and York Collaborations for Leadership in Applied Health Research and Care (CLAHRC) Funded by the NIHR Payment by Results Conference Friday 28th September 2012

NIHR Nine CLAHRCs approx £10million each CLAHRC for Leeds/York/Bradford Addiction Research in Acute Settings (ARiAS) 1 of 5 themes ARiAS outcome measurement 1 of 6 strands CLAHRC research group: Duncan Raistrick, Gillian Tober, Christine Godfrey, Charlie Lloyd, Steve Parrot, Jude Watson, Veronica Dale Co-opted Expert Group convened March 2011: Owen Bowden-Jones, Alex Copello, Ed Day, Eilish Gilvarry, Don Lavoie, Damian Mitchell, Julia Sinclair, John Strang, and Alex Whincup

Summary

• Views and expectations of different stakeholders
• Service user and carers
• Politicians and commissioners
• Practitioners
• Substance use and scales
• Self report
• Biological measures
• Scales EQ5D LDQ CORE SSQ
• Societal Impact Measures
• Further interesting things to find out
• Relationship between measures
• Clinically significant change

Outcome measures answer the question “how do I know this person is getting better?” and they also...

summarise complex information in a clinically meaningful and ‘real world’ way communicate complex information in a clear and simple way using minimal data can be integrated into routine practice in a clinically useful way but.............

......different stakeholders have different ideas of ‘getting better’ ...... the same stakeholders have different needs

• Service users and carers – abstinence
• Politicians and commissioners – costs and benefits
• Practitioners
• Health workers – physical and mental health
• Criminal justice workers – offending behaviour
• Social workers – safeguarding children

......so, there is no ideal measure, just the mix of measures that best suits the purpose.

Service user and carer views on what

• utcomes are important

Agency Type Type of Activity SU F&F

FG1

NHS Treatment 7 4

FG2

NHS/3rd sector DRR 3 2

FG3

NHS/3rd sector Harm reduction

FG4

SMART group Recovery mutual aid 7 n/a

FG5

3rd sector Recovery SU only 7 n/a

FG6

3rd sector Recovery F&F only n/a 6 TOTAL 24 12

Source: unpublished CLAHRC study 2012

Six focus groups (FGs) were held to elicit service user (SU) views and views of Family and Friends (F&F) on what constitutes a good outcome.

Analysis of Focus Groups

FG1 recovery is abstinence, social life, support for family and friends, relationships, lifestyle with partner, being in control, life style changes during recovery process, physical and mental health, confidence FG2 positive environment to maintain abstinence and move away from drugs and alcohol, no good

• utcome – it’s with you for life, not using methadone or buprenorphine, social support, doing

something to fill time, safe accommodation, children back from care, confidence, physical health and appearance FG4 physical and mental health, ability to distract and not act upon cravings, self-worth, energy, not feeling isolated, personal responsibility, living a normal life, engaging in treatment, improved relationships FG5 self esteem, eating healthily, abstinence, sleep, physical health, relationships, social networks, feeling valued, new interests and skills, courses. FG6 employment, safe and supportive social networks, something to do during the day, self-worth, physical and psychological well-being, having a good addiction, not feeling ashamed, financial situation

There was a distinct view that abstinence or perhaps moderation is the first step but the social support to maintain change is all important. Other gains such as health and relationships were seen as good but not the essence of a good outcome.

Source: unpublished CLAHRC study 2012

Substance Use as the Primary Outcome

Justification for substance use as the primary outcome measure.......

• Viewed by service users as most important
• Viewed by careers as most important
• Most convincing for general public and policy

makers

• Substance misuse is the condition
• Correlation with societal costs

Different Ways to Measure Substance Use

Frequency Categories ASSIST & AUDIT Objective Testing blood tests toxicology Episodes

• f Use

OTI Quantity / Frequency MAP & TOP Actual Frequency ASI

Composite measures are restrictive:

• always a compromise – too much/too little detail eg ASI
• typically rater completed eg TOP, ASI, MAP
• item selection bias eg TOP
• predetermined categories not suited to particular needs eg AUDIT
• not easy to update/modify and revalidate eg TOP, ASI

10 20 30 40 50 60 % of sample in drinking volume group Interview year

Abstinent Sensible (<14/21 units) Hazardous (14-35 / 21-50) Heavy (35-70 / 50-100) Very heavy (70+ / 100+)

Birmingham Heavy Drinkers 10yr Follow-up

Uses UNITS of alcohol grouped in ranges

n=259 followed at 10yrs: >50% reported major life events in previous 2yrs: typically health, employment, shift of attitude. 18 known deaths: 3 CVS, 4 liver, 3 cancer, 1 suicide, 2 diabetes. Abstinence increased from 0-10%, weekly units decreased from 90-59 (m) and 60-36 (f)

Source: Rolfe et al.(2009) Report to DH

UK Alcohol Treatment Trial

FREQUENCY and UNITS of alcohol

20 40 60 80 100 Percent of Days Abstinent Baseline 12 Months MET SBNT 5 10 15 20 25 30 Drinks per Drinking Day Baseline 12 months MET SBNT Source: UKATT Research Team, BMJ (2005)

Problems of the substance itself as an

• utcome measure……
• Is it how much (quantity) or how often (frequency) that

matters? How reliably can these be measured?

• Is it the range of substances used or just the presenting

main substance that should be the outcome? Does this include prescribed drugs (eg methadone, diazepam)?

• Are some drugs associated with more harmful routes of

administration than others (eg heroin)?

• Does substance use itself matter or is it the related harms?

Are some harms clearly attributable to particular substances (eg alcohol) some much less so (eg heroin)?

• How to measure it?

72-hour detection window (drugs that can be detected up to 72hrs)

Frequency of drug use Frequency of testing 2x /wk 1x /wk 2x /mth 1x /mth 8x /yr Every day

3±2 7±2 15±10 30±13 46±40

Every other day

4±3 8±3 18±12 35±17 51±50

1x /wk

5±4 11±7 23±18 48±31 71±66

1x /wk

9±9 18±14 40±33 80±64 118±106

2x /mth

19±21 39±35 91±88 160±124 272±260

1x /mth

36±42 71±66 150±141 306±283 560±598

Toxicology Screening as an Indirect Measure

Days (mean ± sd) before a positive test would be expected given different drug use frequency and different testing frequency

Source: Crosby et al., (2003) J Addictive Diseases

Correlations with Societal Costs

example data for units of alcohol and health status

10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60

• ne2one

increase decrease weekly units of alcohol health as % of best possible health

moderate drinking not associated with major health problems health may not recover

• nce damage is done

Correlations with Substance Use.... Baseline Follow-up EQ5D LDQ CORE SSQ EQ5D LDQ CORE SSQ Alcohol freq

• 0.32** 0.48**

0.28**

• 0.08*
• 0.30** 0.62**

0.39**

• 0.20**

Alcohol units

• 0.38** 0.47**

0.25**

• 0.19** -0.30** 0.62**

0.38**

• 0.20**

Heroin freq

• 0.11

0.39** 0.09

• 0.10
• 0.21

0.40** 0.40**

• 0.60**

**p<.001 *p<.01 Source: unpublished CLAHRC clinical sample

but....

• substance use is difficult to measure – LDQ good proxy
• other scales are needed to paint a picture of outcome

Conclusion

There is a scientific and political case for the primacy of substance use as an outcome measure....

Choosing Scales as Secondary Outcome Measures

Types of Outcome Scales

Generic Measures (Health)

Is treatment cost effective? How ill are people with addiction problems compared to other users of health care? How complex are the health problems? What is the illness profile of people with addictions?

Dimension Measures (Addiction)

How severe is the addiction? How difficult is treatment likely to be? How good is

• ne addiction service compared to another? Do problems persist?

Condition Specific Measures (Depression, Pregnancy)

How severe is the specific condition? How do services targeting the condition compare? How effective is treatment for this specific problem?

Personal Goal Measures (Me)

The personal outcome goals agreed with each service user.

Source: Fitzpatrick et al. (1998) Health Technology Assessment

Societal Impact Measures - SIMs

• Treating the individual has benefits at the societal

level – SIMs are service level

• Political interest is in societal costs (a research

exercise) – SIMs are a headline contribution

• SIMs need to be: objective, easy to collect, capable
• f showing variability, reflect the impact of the service

in the fewest possible measures...............

Pregnancy and Parenting i) birth weight ii) child with mother at 12months Hospital In-reach i) A&E attendances ii) in-patient admissions in last 12months Detoxification i) % completing ii) % supervised disulfiram

A Quality Framework for Outcome Scales

Self completion scales are the gold standard Scales evaluated by scoring for:

• Evidence Base (including independent evaluations)
• Psychometric Properties
• Normative Data
• Availability (free and supported by website)
• Ease of Use
• Universality (all substances and all socio-economic

groups)

• Service User Evaluations

Source: NIMHE (2009)

Scales that best match quality criteria...

Evidence (0-2) Number of validation publications Psychometric properties (0-6) Availability (0-2) Practicality (0-3) Universality (0-2) Population norms (0-2)

EQ-5D health

2 tbc 6 2 2 2 2

LDQ dependence

2 3 5 2 3 2 1

CORE-10 mental health

2 2 5 2 3 2 2

SSQ satisfaction

1 1 5 2 3 2 1

APQ problems

2 2 6 2 3

HoNOS mental health

2 tbc 4 2 3 2 tbc

PHQ9 depression

2 tbc tbc 2 3 2 tbc

GAD7 anxiety

2 tbc tbc 2 3 2 tbc

IRS impulsivity

2 5 2 2 2

FMQ family coping

tbc tbc tbc 2 tbc 2

PCQ parenting

1 1 2 2 3 2

EQ5D weighted scores baseline and 3mth

higher score = better health weighted score max = 1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

All Alcohol Heroin & Opiates Methadone Other drug Population 25-35

Baseline Follow up Gen Pop

Source: unpublished clinical data

Frequency of scores at assessment and 3mth

LDQ CORE SSQ

Source: unpublished clinical data

Conclusion

The EQ5D LDQ CORE SSQ are clinically useful, used in routine practice, have predictive value and paint a comprehensive picture....

20 40 60 80 100 5 10 15 20 25 30

Your score now… …this is what happens in treatment

Interesting Things to Find Out

Domains (or components) of Addiction

psychological well being dependence social well being substance use

Factor analysis with #4 factor solution EQ5D LDQ CORE-10 SSQ

F#1 F#2 F#3 F#4 F#1 F#2 F#3 F#4 F#1 F#2 F#3 F#4 F#1 F#2 F#3 F#4 .77 .13 .07

• .09

.10 .80 .25

• .10

.19 .36 .71

• .12

.11

• .04
• .01

.66 .67

• .02

.15

• .08

.12 .81 .24

• .15
• .15
• .01

.15

• .40
• .01
• .05
• .04

.70 .57 .28 .31

• .15

.12 .80 .27

• .10

.02 .26 .44

• .22
• .15
• .19
• .03

.57 .68 .22 .08

• .07

.11 .83 .20

• .11

.04 .19 .60

• .14
• .05
• .22
• .06

.59 .25 .31 .69

• .12

.18 .65 .20

• .11

.28 .25 .75

• .10
• .17
• .13
• .30

.56 .15 .75 .18

• .13

.34 .14 .57

• .20
• .11
• .07
• .30

.60 .04 .75 .25

• .08

.26 .30 .55

• .11
• .06
• .05
• .20

.64 .04 .66 .22

• .12

.22 .31 .77

• .22
• .02
• .11
• .25

.57 .09 .78 .22

• .12

.20 .30 .73

• .23

.15 .74 .35

• .13

.27 .22 .69

• .17

8% of variance 23% of variance 17% of variance 9% of variance

Relationship between EQ5D LDQ CORE SSQ

LDQ and CORE close to each other. Predicts if dependence is treated then mental health improves. SSQ is most separated. Predicts it will change most differently to other measures. EQ5D is generic but independent and between the other

• measures. Has less

influence than LDQ on mental health more on social satisfaction.

Clinically Significant Change - ‘Gold Standard’

Reliable Change Score Well Functioning Population LDQ >= 4 < 12 CORE-10 >= 6 < 14 SSQ >= 4 > 10

Source: CLAHRC submitted

Jacobson et al. (1999) proposed that in order to take account of baseline scores and measuring error, clinically significant change should a) be statistically reliable b) end scores be in a well functioning population range

retest e

r 1 S S

1

 

2 e diff

) 2(S S 

RC = reliable change 95% probability if RC >=1.28 Sdiff = standard error of difference between means of LDQ scores Se = standard error of measurement of LDQ S1 = standard deviation of mean1 rretest = test/retest reliabilty of LDQ

Clinically Significant Change - example

10 20 30 40 50 60 70 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 LDQ scores

reliable change >=4 if pushed into well functioning population range then Clinically Significant Change achieved well functioning population is 2standard deviation above general population mean LDQ<12

Reliable (RC) and clinically significant (CS) change (n-925).... n=925

LDQ % CORE % SSQ % RC improved 61.0 46.0 36.7 RC worse 4.6 5.7 12.0 Too small for RC 14.3 6.1 5.2 CS improvement 50.1 30.5 31.9 clinically significant change at 3mth (drinking n=396)..... CS change % Yes No Yes No Yes No Drinking 22.0 31.1 13.4 39.6 16.7 36.4 Abstinent 35.4 11.6 24.2 22.7 17.7 29.3

P<.001 P<.001 n.s. 95% probability

• f real

change social situation most difficult to recover abstinence associated with the most

• CSC. LDQ

works across drinking

• utcomes

CSC is a tough test of improvement The GOLD standard

Thank you. Questions?

Conclusion

• multiple measures are needed to answer different stakeholder

questions

• always include substance misuse
• avoid composite measures – tailor to suit the agency
• choose scales that meet Quality Framework criteria
• choose clinically useful measures
• add Societal Impact Measures
• check that the package is ethical and easy for routine use