Measurement In Implementation Science: Status and Directions Bryan - - PowerPoint PPT Presentation

1

Measurement In Implementation Science: Status and Directions

Bryan J. Weiner, Ph.D. University of Washington

2

What Do We Know?

3

What Is the Current State?

• Measures are unevenly distributed
• Measures have dubious content validity
• Measures have unknown or poor quality
• Measures are not practical (pragmatic)

4

What Does the Current Look Like?

5

How Much Psychometric Information is Out There?

Construct Internal Consistency Structural Validity Predictive Validity N % N % N % Acceptability 36 72 13 26 11 22 Adoption 8 42 7 37 5 26 Appropriateness 2 29 2 29 1 14 Cost Feasibility 1 13 1 13 Penetration 1 25 1 25 1 25 Sustainability 3 38 3 38 1 13

6

How Much Psychometric Strength Do We See?

Construct Internal Consistency Structural Validity Predictive Validity Median SD Median SD Median SD Acceptability 3.71 0.70 3.46 0.66 2.32 1.31 Adoption 3.50 1.07 2.50 1.90 3.00 0.35 Appropriateness 3.50 0.71 0.50 0.00

• Cost
• Feasibility
• Penetration
• Sustainability

3.33 0.58 2.33 1.53

7

Are There Some Good Measures?

Construct Measure Source

Attitudes Evidence-Based Practice Attitudes Scale Aarons et al 2004 Leadership Implementation Leadership Scale Aarons et al 2014 Readiness Organizational Readiness for Implementing Change Shea et al 2014 Implementation Climate Implementation Climate Implementation Climate Scale Jacobs et al. 2014 Earhart et al. 2015 Implementation Process Stages of Implementation Completion Chamberlain et al 2011 Saldana et al 2014 Inner Setting Constructs Culture, Implementation Climate, Learning Climate, Leadership Engagement, Available Resources Fernandez et al 2018

8

Can We Speed Things Up?

Acceptability of Intervention Measure (AIM) Intervention Appropriateness Measure (IAM) Feasibility of Intervention Measure (FIM)

9

What Did We Do?

10

What Did We Accomplish?

Psychometric Property Content validity  Discriminant content validity  Reliability  Structural validity  Structural invariance  Known-groups validity  Predictive validity Underway Responsiveness to change 

• Directly measure construct
• Show some discriminant validity
• Could be used for EBPs
• r implementation strategies
• Item referents can be tailored

(e.g., appropriateness)

• Short in length

Acknowledgement: R01MH106510

11

What Are Some Take-Aways?

Rapid-cycle development and testing is possible Better measures are needed in implementation science Poor measurement practices abound More attention is needed to theory-based measurement

12

Articles Cited

• Aarons GA. (2004). Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-Based Practice

Attitude Scale (EBPSAS). Ment Health Serv Res. 2004 Jun;6(2):61-74.

• Aarons GA, Ehrhart MG, Farahnak LR. The Implementation Leadership Scale (ILS): development of a brief measure of unit level

implementation leadership. Implement Sci. 2014 Apr 14;9:45.

• Jacobs SJ, Weiner BJ, Bunger AC. Context matters: measuring implementation climate among individuals and groups. Implement
• Sci. 2014 Apr 17;9:46.
• Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational contexts of EBP implementation: the development and validity

testing of the Implementation Climate Scale (ICS). Implement Sci. 2014 Oct 14;9:157.

• Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. Organizational readiness for implementing change: a psychometric

assessment of a new measure. Implement Sci. 2014 Jan 10;9(1):7.

• Chamberlain P, Brown CH, Saldana L. Observational measure of implementation progress in community based settings: the Stages
• f Implementation Completion (SIC). Implement Sci. 2011 Oct 6;6:116.
• Saldana L. (2014). The stages of implementation completion for evidence-based practice: protocol for a mixed methods study.

Implement Sci. 2014 Apr 5; 9:43.

• Fernandez MF, Walker TJ, Weiner BJ, Calo WA, Lian S, Risendal B, Friedman DB, Tu SP, Williams RS, Jacobs S, Herrmann AK,

Kegler MC. Developing measures to assess constructs from the inner setting domain of the Consolidated Framework for Implementation Research. Implement Sci. 2018 Mar 27; 13:52.

13

THANK YOU!

• Bryan J. Weiner, Ph.D.
• University of Washington

@bjweiner