Core Outcome Measures in Effectiveness Trials - - PowerPoint PPT Presentation

Core Outcome Measures in Effectiveness Trials http://www.liv.ac.uk/nwhtmr/comet/comet.htm

Acknowledgements

• Medicines for Children Research Network and

National Institute for Health Research

• COMET Management Group

– Doug Altman, Jane Blazeby, Mike Clarke, Paula Williamson, Elizabeth Gargon (Project coordinator)

• COMET Advisors

– Peter Tugwell, Maarten Boers, Rosalind Smyth, Ian Sinha, Angus McNair, Declan Devane, Tracey Howe

Outline

• Identify problems with outcomes in clinical

trials

• Core outcome sets

– Rationale – Examples – Delphi process

• COMET Initiative

Why are outcomes important?

• Interventions are compared in RCTs by

measuring differences in patient outcomes between the groups

• Selection of appropriate outcomes is crucial
• ‘Clinical trials are only as credible as their
• utcomes’ (Tugwell 1993)

Definitions

• Domain:

– Broad aspects of the effects of illness on a patient

• Outcomes:

– Measurable variables within domains

• Outcome measurement tools

– Tools for measuring an outcome

• What to measure? How to measure it?

Which domains to measure?

• Healthcare interventions can have various

effects

– Disease activity – Disease damage – Functional status – QoL – Health resource utilisation – Adverse effects of therapy

• Which of these should be measured?

Antibiotics for urine infection in children

Antibiotics for urine infection in children

How to measure domains

• Various outcomes within each domain
• Various ways of measuring these outcomes

Treating upper arm fractures in adults

Comparison: ARV vs placebo Outcome: HIV infection

STUDY NUMBER OF PATIENTS Birth 4-8 weeks 3-4 months 6 months 12 months 18 months DITRAME 431 RETRO-CI 281 PETRA 1457 PACTG076 477 LIMPONGS. 182 THAI CDC 423 total 3251 NUMBER (%) OF PATIENTS FROM ALL TRIALS NOT INCLUDED AS OUTCOME NOT MEASURED AT TIMEPOINT 2547 (78%) 659 (20%) 2539 (78%) 2638 (81%) 2820 (87%) 886 (27%)

Problems with outcomes in six important Cochrane reviews

5 most accessed in 2009 *:

• Preventing childhood obesity
• Treating childhood obesity
• Promoting activity in children
• Preventing falls in the elderly
• Preventing Type 2 DM

Most cited in 2009 *:

• Nicotine replacement therapy
• Tovey D. Impact of Cochrane Reviews [editorial]. The Cochrane Library 2010

(7 July)

“We sought data for rate of falls, number of people falling, and number of people sustaining a fracture. However, few studies provided fracture data.”

(Preventing falls in the elderly)

“No study reported relevant data on diabetes and cardiovascular related morbidity, mortality and quality of life..”

(Preventing Type 2 DM)

“The studies … varied greatly in intervention design, outcome measurements and methodological quality.” (Preventing childhood obesity) “Appropriate short- and long-term outcomes need to be defined for children and youth at various weight levels, rather than using conventional

• r adult-oriented outcomes.”

(Treating childhood obesity)

“The studies were heterogeneous in terms of study design, quality, target population, theoretical underpinning, and outcome measures, making it impossible to combine study findings using statistical methods..”

(Promoting activity in children)

“Definitions of abstinence varied considerably ... In five studies it was unclear exactly how abstinence was defined.”

(Nicotine replacement therapy)

Problems with outcomes in MS reviews

• 30 MS reviews in the Cochrane Library
• None specifically paediatric

– 14 did not specify age in the inclusion criteria, although looked like adult studies – 8 included patients of all ages, although looked like adult studies – 7 stated adult patients only – 1 included adults and children but no studies met inclusion criteria

How often did reviewers comment about heterogeneity of outcomes?

• 26 out of 30 reviews (87%)

– unable to combine/pool data for quantitative analysis – variation in definition of outcome/outcome criteria – inadequate and inconsistent reporting of outcome data

• outcome reporting bias

– time point of outcome measurement – inappropriate outcome measures used – large variety of outcome measures used

Outcome reporting bias

• Definition: Selection of a subset of the original

recorded outcomes, on the basis of the results, for inclusion in publication

• Statistically significant outcomes more likely

to be fully reported: OR 2.2 to 4.7 (Dwan et al, 2008)

ORBIT: key messages (Kirkham et al 2010)

• ORB suspected in at least one trial in >33% of 283

reviews

• 42 significant meta-analyses

– 8 (19%) would not have remained significant – 11 (26%) would have overestimated the treatment effect by > 20%

What’s the solution?

• Core outcome sets

– an agreed standardised set of outcomes that should be measured and reported, as a minimum, in all clinical trials in specific areas of health or health care

• Relevant within routine clinical practice
• Advantages

– less heterogeneity between trials – reduced risk of ORB – more likely to measure appropriate outcomes

Cancer trials

• In 1977, World Health Organisation convened a

meeting to discuss “Standardization of Reporting Results of Cancer Treatment”, Turin

• In 1979, meeting of 30 representatives from

international cooperative groups working on trials in cancer, Brussels

• In 1979, WHO Handbook for Reporting Results of

Cancer Treatment

Items

• Acute toxicity
• Chronic or late toxicity
• Partial response (>50% decrease in tumour load)
• Complete response (100% disappearance)
• Date of first recurrence
• Date of death

• www.omeract.org
• Trials 2007 8:38

Appropriate outcomes

• Patient involvement
• Impact

– rheumatology (OMERACT): fatigue – chronic pain (IMMPACT): expansion of previously proposed core outcome domains – paediatric asthma (parents and children)

Paediatric-specific example

• Does a COS exist for trials of regular paediatric

asthma therapies?

• Is there heterogeneity between trials with regard to
• utcomes?
• How should we develop a COS?

PLoS Medicine 2008 5(4):e96

The Delphi technique

• Structured technique for reaching consensus
• Panel of ‘experts’
• Sequential questionnaires
• Anonymised
• Feedback after each round
• Avoids problems of face-to-face interaction
• Enables use of large panel

Using the Delphi technique to develop core outcomes

• No guidance
• Aim: to systematically review studies that used the

Delphi technique to determine which outcomes to measure in trials

• 15 studies (8 rheumatology)
• Variations in methodology:

– Group composition and size – How potential outcomes were initially identified – Structure of the Delphi

• Variable reporting quality

Potential areas of bias

• Group composition

– Mainly researchers – Patients/families seldom involved

• One panel or separate panels?
• How outcomes identified at start of process

– Mainly considered outcomes that were already measured in clinical trials – Seldom asked open questions

• Attrition of participants

Khan, F. and J. F. Pallant (2007)

• Preliminary ICF published studies
• Review of evidence base and literature
• Delphi exercise with physicians and allied

health professionals

– 3 rounds

Khan, F. and J. F. Pallant (2007)

• The brief core set is a list of categories that

should be recorded in every clinical study in which the effect of an intervention on patients with MS is to be determined.

• This set is preliminary

– Needs to include representatives from MS

• rganisations, consumer groups, and importantly

patients with MS – Needs international input

COMET Initiative

• An international network
• Management group and advisors
• Facilitate and promote development and application of core outcome sets
• Work on core outcome sets in over 40 areas e.g.
• cancer
• rheumatology (OMERACT)
• chronic pain (IMMPACT)
• maternity care (WOMBAT)
• Collate and maintain relevant resources in a publically available

searchable database

• does a COS exist for trials of regular paediatric asthma therapies?

COMET Initiative

• ‘What’ to measure
• Provide guidance on methods for developing core outcome sets
• Little or no patient involvement – examples where patients identified an
• utcome important to them that has not been previously measured
• Develop reporting standards for such studies
• e.g. studies using Delphi methods
• ‘How’ to measure (validity, reliability, feasibility)
• PROMIS, http://proqolid.org/
• COSMIN
• TREAT-NMD
• Musculoskeletal
• Paediatrics

What next for COMET?

• Website and database development
• Guidance on developing a core outcome set
• 2nd COMET Meeting, July 11-12, Bristol, UK

2nd COMET Meeting

• 11th-12th July 2011
• Ashton Court Mansion - Bristol, UK
• Registration now open
• www.cometmeeting2011.org

References

• Sinha I, Jones L, Smyth RL, Williamson PR. A Systematic Review of Studies That Aim to

Determine Which Outcomes to Measure in Clinical Trials in Children. PLoS Med 2008. 2008; 5(4): e96

• Sinha I, Williamson PR, Smyth RL. Outcomes in Clinical Trials of Inhaled Corticosteroids for

Children with Asthma Are Narrowly Focussed on Short Term Disease Activity. PLoS One. 2009; 4(7): e6276.

• Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan A-W, et al. Systematic Review of the Empirical

Evidence of Study Publication Bias and Outcome Reporting Bias. PLoS ONE. 2008;3(8):e3081

• Kirkham JJ, Dwan KM, Altman DA, Gamble C, Dodd S, Smyth RMD, Williamson PR (2010) The

impact of outcome reporting bias in randomised controlled trials on a cohort of systematic

• reviews. BMJ 340: c365, doi: 10.1136/bmj.c365
• Smyth RMD, Kirkham J, Jacoby A, Altman DG, Gamble CL, Williamson PR. Frequency and

reasons for outcome reporting bias in clinical trials: interviews with trialists. BMJ, accepted

• Sinha I, Smyth RL, Williamson PR. Recommendations for the conduct and reporting of studies

which use the Delphi process to determine which outcomes to measure in clinical trials. Submitted