Measurement In Implementation Science: Status and Directions Bryan - PowerPoint PPT Presentation

1 Measurement In Implementation Science: Status and Directions Bryan J. Weiner, Ph.D. University of Washington

What Do We Know? 2

What Is the Current State? 3 Measures are unevenly distributed • • Measures have dubious content validity • Measures have unknown or poor quality Measures are not practical (pragmatic) •

What Does the Current Look Like? 4

How Much Psychometric Information is Out There? 5 Construct Internal Consistency Structural Validity Predictive Validity N % N % N % Acceptability 36 72 13 26 11 22 Adoption 8 42 7 37 5 26 Appropriateness 2 29 2 29 1 14 Cost 0 0 0 0 0 0 1 13 1 13 0 0 Feasibility Penetration 1 25 1 25 1 25 Sustainability 3 38 3 38 1 13

How Much Psychometric Strength Do We See? 6 Construct Internal Consistency Structural Validity Predictive Validity Median SD Median SD Median SD 3.71 0.70 3.46 0.66 2.32 1.31 Acceptability 3.50 1.07 2.50 1.90 3.00 0.35 Adoption 3.50 0.71 0.50 0.00 Appropriateness - - Cost - - - - - - Feasibility - - - - - - Penetration - - - - - - 3.33 0.58 2.33 1.53 - - Sustainability

Are There Some Good Measures? 7 Construct Measure Source Attitudes Evidence-Based Practice Attitudes Scale Aarons et al 2004 Leadership Implementation Leadership Scale Aarons et al 2014 Readiness Organizational Readiness for Implementing Change Shea et al 2014 Implementation Implementation Climate Jacobs et al. 2014 Climate Implementation Climate Scale Earhart et al. 2015 Implementation Stages of Implementation Completion Chamberlain et al 2011 Process Saldana et al 2014 Inner Setting Culture, Implementation Climate, Learning Climate, Fernandez et al 2018 Constructs Leadership Engagement, Available Resources

Can We Speed Things Up? 8 Acceptability of Intervention Measure (AIM) Intervention Appropriateness Measure (IAM) Feasibility of Intervention Measure (FIM)

What Did We Do? 9

What Did We Accomplish? 10 • Psychometric Property Directly measure construct Content validity  • Show some discriminant validity Discriminant content validity  • Could be used for EBPs Reliability  or implementation strategies Structural validity  • Item referents can be tailored Structural invariance  (e.g., appropriateness) Known-groups validity  • Short in length Predictive validity Underway Responsiveness to change  Acknowledgement: R01MH106510

What Are Some Take-Aways? 11 Better measures are needed in implementation science Poor measurement practices abound More attention is needed to theory-based measurement Rapid-cycle development and testing is possible

Articles Cited 12 • Aarons GA. (2004). Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-Based Practice Attitude Scale (EBPSAS). Ment Health Serv Res. 2004 Jun;6(2):61-74. • Aarons GA, Ehrhart MG, Farahnak LR. The Implementation Leadership Scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci. 2014 Apr 14;9:45. • Jacobs SJ, Weiner BJ, Bunger AC. Context matters: measuring implementation climate among individuals and groups. Implement Sci. 2014 Apr 17;9:46. • Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational contexts of EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). Implement Sci. 2014 Oct 14;9:157. • Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. Organizational readiness for implementing change: a psychometric assessment of a new measure. Implement Sci. 2014 Jan 10;9(1):7. • Chamberlain P, Brown CH, Saldana L. Observational measure of implementation progress in community based settings: the Stages of Implementation Completion (SIC). Implement Sci. 2011 Oct 6;6:116. • Saldana L. (2014). The stages of implementation completion for evidence-based practice: protocol for a mixed methods study. Implement Sci. 2014 Apr 5; 9:43 . • Fernandez MF, Walker TJ, Weiner BJ, Calo WA, Lian S, Risendal B, Friedman DB, Tu SP, Williams RS, Jacobs S, Herrmann AK, Kegler MC. Developing measures to assess constructs from the inner setting domain of the Consolidated Framework for Implementation Research. Implement Sci. 2018 Mar 27; 13:52.

13 THANK YOU! • Bryan J. Weiner, Ph.D. • University of Washington @bjweiner