Three Key Analytic Processes That Emerged While Using the - - PowerPoint PPT Presentation

Three Key Analytic Processes That Emerged While Using the Consolidated Framework for Implementation Research (CFIR) to Evaluate Broad-Scale System Change Jennifer N. Hill, MA

Implementation Research Scientist Department of Veterans Affairs Center of Innovation for Complex Chronic Healthcare (CINCCH) Combating Antimicrobial Resistance through Rapid Implementation of Available Guidelines and Evidence (CARRIAGE) QUERI 9th Annual Conference on the Science of Dissemination and Implementation December 14, 2016

VETERANS HEALTH ADMINISTRATION

Disclosures and Funding Information

• The views expressed in this presentation are those of the authors and do not

necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.

• This article is based on evaluations funded by the Department of Veterans Affairs

Office of Patient-Centered Care & Cultural Transformation and the VA Health Services Research & Development Quality Enhancement Research Initiative (PCE 13-001, PI: Bokhour; PCE 13-002, PI: LaVela).

1

VETERANS HEALTH ADMINISTRATION

Co-authors

• Sara M. Locatelli, PhD - Edward Hines Jr. VA Hospital, Center for Evaluation of Practices and

Experiences of Patient-Centered Care (CEPEP), Center of Innovation for Complex Chronic Health Care (CINCCH)

• Barbara G. Bokhour, PhD - Center for Healthcare Organization and Implementation Research

(CHOIR), ENRM Veterans Affairs Medical Center; Center for Evaluating Patient-Centered Care (EPCC); Boston University School of Public Health

• Gemmae M. Fix, PhD - Center for Healthcare Organization and Implementation Research

(CHOIR), VA Health Services Research and Development Service, Boston University School of Public Health; Evaluating Patient-Centered Care (EPCC)

• Jeffrey Solomon, PhD - Evaluating Patient-Centered Care (EPCC)
• Nora Mueller, MS - Center for Healthcare Organization and Implementation Research (CHOIR),

ENRM Veterans Affairs Medical Center; Center for Evaluating Patient-Centered Care (EPCC)

• Sherri L. LaVela, PhD, MPH, MBA - Edward Hines Jr. VA Hospital, Center for Evaluation of

Practices and Experiences of Patient-Centered Care (CEPEP); Center of Innovation for Complex Chronic Health Care (CINCCH)

2

VETERANS HEALTH ADMINISTRATION

Background: Theories and Frameworks for Evaluation

• Theories and frameworks describe and prescribe aspects of an evaluation, rooted

in the needs or requirements of the customer and the purpose of the inquiry.

• These include activities or strategies, methods choices, and the responsibilities of

and products to be provided by the evaluators [1].

• The Consolidated Framework for Implementation Research (CFIR)

– Offers a comprehensive taxonomy of constructs related to the intervention, inner and outer settings, characteristics of individuals, and implementation process [2]. – Primarily has been used to evaluate implementation of single, discrete interventions or programs [3-5] – May also be useful for evaluating broad-scale programs implemented by large, integrated healthcare systems.

3

VETERANS HEALTH ADMINISTRATION

Background: The Need for Evaluating a Broad-Scale System Change in VA

• The Department of Veterans Affairs (VA) has committed to transforming from a

problem-based, disease-focused system of care to one that is patient-centered.

• This commitment was solidified with the creation of the Office of Patient-Centered

Care and Cultural Transformation (OPCC&CT) in 2010 [6]. The office:

– Established four Centers of Innovation (COIs) or “learning laboratories” where PCC innovations are implemented and lessons are gleaned for future large system rollouts. – They sought to understand how these COIs were implementing PCC and, – Engaged health services researchers to evaluate its implementation; given the size and scope of the transformation, two evaluation teams shared this task.

• The objective of this presentation is to discuss three key analytic processes that

emerged while using CFIR to evaluate a broad-scale system change.

4

VETERANS HEALTH ADMINISTRATION

Selection of CFIR and Design of the Evaluation

• As part of the application process, each evaluation team had independently selected

CFIR to plan and structure their evaluations base on its comprehensiveness as well as the flexibility offered in recommendations of its use [7].

• While finalizing the constructs to be evaluated team used the “menu of constructs”

process [only constructs essential to the evaluation were included].

• The selection of these ‘essential constructs’ was informed by the scope of the

evaluation and the critical questions and goals of key leadership.

• Operational definitions [8] were developed for the selected constructs and the

framework was integrated into the evaluation design, data collection, and analysis.

5

VETERANS HEALTH ADMINISTRATION

Examples of Selected Constructs and Rationales

6

Construct [Selection of Construct] - Rationale for Selection Domain: Intervention Characteristics Intervention Source [Evaluation Team] - The intervention was ongoing, evaluation team wanted to capture historical information. Adaptability [Evaluation Team] - Document adaptations, especially given the broad scope. Complexity [Evaluation Team] - Perceived complexity of the PCC cultural transformation. Domain: Outer Setting Patient Needs & Resources [OPCC&CT and Evaluation Team] - Patient involvement in the transformation. Domain: Inner Setting Climate\Culture [OPCC&CT and Evaluation Team]- Receptivity to the transformation. Domain: Process Engaging: Staff [OPCC&CT and Evaluation Team] - How and why staff are engaged Reflecting & Evaluating [OPCC&CT and Evaluation Team] -Site-level processes for tracking progress

VETERANS HEALTH ADMINISTRATION

Methods: Study Participants and Recruitment

• Senior management and PCC coordinators at each facility recommended potential

participants to the evaluation teams.

• To reduce coercion, these individuals were invited to participate via an

informational email which offered them the opportunity to contact evaluation team.

• Interviewees were told their responses would be used by OPCC&CT to guide future

program efforts including addressing program barriers and sharing of best practices.

• Interviews were conducted either in-person during site visits or over the telephone

if the participant was unavailable during the site visit.

7

VETERANS HEALTH ADMINISTRATION

Methods: Qualitative Analysis

• A mixed deductive-inductive [9,10] approach to coding was used to analyze data from

the interviews.

• Deductive coding was:

– Guided by CFIR with a structured analytical tool to facilitate rapid qualitative analysis – To investigate implementation processes and successes/failures (deductive) and barriers and facilitators tied to the local context

• Inductive coding was used to:

– Capture themes not represented in CFIR to ensure coding was reflective of the data

8

VETERANS HEALTH ADMINISTRATION

Methods: Process of Synthesis for Operations Partners

• The final step of the analysis involved synthesizing these themes into key domains

impacting implementation of PCC and cultural transformation.

• This process involved:

(1) conducting additional analyses on data within key domains to define how and why it was salient (2) developing recommendations that could be utilized by leadership to enhance implementation of the program.

• These analytical processes undertaken were tracked by the team as they were

considered essential components of utilizing CFIR for this broad scale evaluation.

• All key analytic processes identified by the team and the documented aspects of those

processes are presented in the results.

9

VETERANS HEALTH ADMINISTRATION

Results: Overview

• While using CFIR to evaluate these broad-scale programmatic changes, 3 key

analytic processes were identified: (1) the creation of adapted definitions for the CFIR constructs to account for its application to the broad-scale evaluation (2) the mixed deductive-inductive coding process demonstrated the flexibility of CFIR for complex evaluation in the emergence of additional CFIR constructs and the several new key themes from the co-occurring inductive thematic coding. (3) the rapid analysis and synthesis of the data into key domains impacting implementation of PCC&CT to develop recommendations to support enhancement

• f implementation and expansion opportunities for the program .

10

VETERANS HEALTH ADMINISTRATION

Results: Adapted Definitions

• As a first step, the evaluation team reviewed the CFIR domains and constructs and

developed adapted definitions based on the study context, including: (1) the broad scope of the PCC intervention(s) (2) the broad-scale culture change targeted (3) the input and existing knowledge of the operations partner OPCC&CT (4) the goals of the evaluation including assessing what had already occurred and what was currently in progress.

11

VETERANS HEALTH ADMINISTRATION

Results: Adapted Definitions (cont.)

• Some definitions required more adaptation than others.
• Ex. Domain/construct “Intervention Characteristics/Complexity” or “Inner

Setting/Culture” did not require adaptation of the definition

• However the questions associated with measuring the constructs had to be broader than

those typically associated with a single-intervention evaluation.

• Other domain constructs required adaptation of the definitions to fit the goals of

the evaluation and the needs of the operations partner.

• Ex. Domain/construct “Outer Setting/Patient Needs & Resources” was adapted:
• Standard short definition “the extent to which patient needs, as well as barriers and

facilitators to meet those needs, are accurately known and prioritized by the organization”

• Adapted definition “Identified patient needs, processes used to identify them, barriers and

facilitators with meeting needs, and strategies for engaging patients to address needs.”

12

VETERANS HEALTH ADMINISTRATION

Results: Additional Examples of Adapted Definitions

13

CFIR Construct Adapted Definitions Domain: Intervention Characteristics Intervention Source History of PCC-related program(s) or practice(s) and perceived source of the initiative Evidence Strength/Quality Perception of intervention patient-centeredness Domain: Outer Setting Patient Needs & Resources Identified patient needs, processes used to identify them, barriers and facilitators associated with meeting needs and strategies for engaging patients to identify ways to address them Domain: Inner Setting Leadership Engagement Commitment, involvement, and accountability of leaders and managers Domain: Characteristics of the Individual Self-efficacy Individual belief in capabilities to achieve implementation goals Domain: Process Reflecting & Evaluating Steps to evaluate implementation and patient outcomes

VETERANS HEALTH ADMINISTRATION

Results: Emergence of CFIR Constructs (Deductive) and New Thematic Codes (Inductive) Overview

• Interview data revealed that out of the selected constructs targeted in the interview

guide, all 19 (100%) were identified as important influences on the implementation

• f PCC.
• Several interview questions encouraged longer narrative type answers such as:

What are the key elements for care to be patient-centered from your perspective?” and “Tell me a little bit about the history of transforming the organization to become more patient-centered.”

• These types of questions, along with follow-up and probe questions resulted in the

emergence of additional CFIR constructs.

• In fact, another 16 CFIR constructs emerged when using deductive coding with the

CFIR structured analytical tool.

14

VETERANS HEALTH ADMINISTRATION

Results: Examples (Emergence of Additional CFIR Constructs)

15

5 CFIR Domains Pre-Planned Constructs with Direct Interview Questions Additional Constructs Emerging from Interviews

• II. Outer Setting

Patient Needs Cosmopolitanism Peer Pressure External Policies and Incentives

• III. Inner Setting

Culture or Implementation Climate Tension for Change Relative priority Organizational Rewards and Incentives Goals and Feedback Leadership Engagement Available Resources Structural Characteristics Networks and communications Compatibility Learning Climate Access to Knowledge and Information

• IV. Characteristics of

Individuals Knowledge and Beliefs of the Intervention Self-efficacy Individual Stage of Change Individual Identification with Organization

VETERANS HEALTH ADMINISTRATION

Results: Importance of the Mixed Deductive/Inductive Approach

• The mixed deductive-inductive approach to coding enabled the team to utilized

thematic coding (inductive) to create codes for additional themes that: (1) were not fully represented by a CFIR construct (2) provided context-specific details (3) offered advantages for organization of ideas

16

VETERANS HEALTH ADMINISTRATION

Results: Importance of the Mixed Deductive/Inductive Approach

17

Construct/Theme Rationale for Selection Definition and/or Examples of Themes Not fully-represented by CFIR Key Strategies OPCC&CT and Evaluation Team: Innovative or best practices for ongoing reporting and dissemination to the field Key strategies for implementing PCC (i.e. What’s worked well in implementing PCC) Provided Context-Specific Detail Role in VA Evaluation Team: Understand implications of dual roles related to implementation Official VA title and general roles and responsibilities outside of PCC (if applicable) Offered Advantages for Organizing Ideas PCC Barriers OPCC&CT and Evaluation Team: Identification of factors influencing implementation for ongoing reporting and dissemination to the field Barriers related to implementing PCC (dual coded with other CFIR/OTM constructs) PCC Facilitators Facilitators related to PCC (dual coded with other CFIR/OTM constructs)

VETERANS HEALTH ADMINISTRATION

Results: Rapid and Actionable Feedback (Overview)

• The availability of the pre-defined constructs allowed for identification of factors

influencing implementation in an organized and easily accessible way.

• The evaluation team was able to deliver a methodologically sound, prompt analysis

and development of timely, meaningful recommendations to operational partner. – 107 interviews were conducted, transcribed, and analyzed over a period of approximately five months. – Recommendations that could be used to facilitate development of strategies and processes to support future implementation efforts which was delivered in month 6.

18

VETERANS HEALTH ADMINISTRATION

Results: Rapid and Actionable Feedback: Recommendations 1-3

• Patient needs and resources, engaging patients: Successes were often attributed to

engagement of patients to ensure their voices were being represented in efforts.

– Action: Offer many opportunities for both formal (e.g., councils and advisory groups) and informal (e.g., point-of care surveys, town halls) feedback for patients

• Leadership engagement: Facilities who had leaders that were both vocal and visibly

supportive of PCC efforts helped facilitate a culture of adoption of PCC values.

– Action: Establish a leadership group in the facility responsible for and embodying the tenants of the transformation

• Structural characteristics, hiring challenges for unique positions (e.g. massage

therapist or acupuncturist): Facilities supported the cross-training of nurses, therapists, and others in PCC interventions (e.g., mindfulness, healing touch)

– Action: Demonstrate flexibility and efforts for integration of PCC into existing practices that are consistent with transformation efforts

19

VETERANS HEALTH ADMINISTRATION

Results: Rapid and Actionable Feedback: Recommendations 4-6

• Knowledge & beliefs, clarify expectations: Lack of clarity in individual roles and

measures of success diminished motivation for and urgency of implementation.

– Action: Provide examples of individual-level PCC behaviors/actions and create performance measures aligned with them to promote accountability

• Engaging staff: Encouraging staff to share their visions and ideas was noted as

something that supported positive cultures, receptive to implementation.

– Action: Foster creativity among staff to facilitate staff involvement in and promotion of PCC

• Champions and/or formally appointed champions: Use of enthusiastic peer champions

to express support, share evidence, and recommend strategies were essential.

– Action: Leverage mid-level managers in the field as a connection to support translation

• f PCC from senior-management to frontline workers

20

VETERANS HEALTH ADMINISTRATION

Discussion

• This as a study that:

– Used CFIR to understand something quite distinct from the implementation of a single intervention or evidence based practice. – Assessed the spread of a concept, PCC, which involves multiple complementary and sometimes overlapping activities aimed at a similar outcome. – Is among a small number of studies to use CFIR for guiding design, data collection, coding, analysis and to evaluate a broad-scale system change.

21

VETERANS HEALTH ADMINISTRATION

Discussion (cont.)

• This evaluation differs from other studies using CFIR to evaluate discrete interventions

[3-5,7] in which findings are nearly exclusively tied to the framework because evaluation of this broad-scale change required testing and exercising CFIR’s flexibility.

• This evaluation builds upon the work of Damschroder and colleagues who used CFIR

to evaluate a large-scale weight management program – in which they chose not to do parallel inductive coding [7].

• The rapid analysis proved critical for OPCC&CT as they quickly operationalized the

findings and disseminated a document to the field and their stakeholders .

– Lessons from the Field – Operational Tactics for Implementing Patient Centered Care and Cultural Transformation which proposed “operational tactics” or steps to addressing findings from the white paper described in OPCC&CTs Annual Report [26].

22

VETERANS HEALTH ADMINISTRATION

Contact information

If you have any questions or comments please feel free to email me at: Jennifer.Hill3@va.gov

23

VETERANS HEALTH ADMINISTRATION

References

1. Coryn CL, Noakes LA, Westine CD, Schroter DC: A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation 2011, 32: 199-226. 2. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009, 4: 50. 3. Damschroder LJ, Hagedorn HJ: A guiding framework and approach for implementation research in substance use disorders

• treatment. Psychol Addict Behav 2011, 25: 194-205.

4. Cabassa LJ, Gomes AP, Lewis-Fernandez R: What would it take? Stakeholders' views and preferences for implementing a health care manager program in community mental health clinics under health care reform. Med Care Res Rev 2015, 72: 71- 95. 5. Kilbourne AM, Abraham KM, Goodrich DE, Bowersox NW, Almirall D, Lai Z et al.: Cluster randomized adaptive implementation trial comparing a standard versus enhanced implementation intervention to improve uptake of an effective re-engagement program for patients with serious mental illness. Implement Sci 2013, 8: 136. 6. Gaudet T: Transforming the Veterans Health Administration System: Personalized, proactive, and patient-centered care. Archives of Agronomy and Soil Science 2014, 20: 11-15. 7. Damschroder LJ, Lowery JC: Evaluation of a large-scale weight management program using the consolidated framework for implementation research (CFIR). Implement Sci 2013, 8: 51. 8. Kirk MA, Kelley C, Yankey N, Birken SA, Abadie B, Damschroder L: A systematic review of the use of the Consolidated Framework for Implementation Research. Implement Sci 2016, 11: 72. 9. Van Gorp B: The constructionist approach to framing: Bringing culture back in. Journal of communication 2007, 57: 60-78. 10. Druetz T, Kadio K, Haddad S, Kouanda S, Ridde V: Do community health workers perceive mechanisms associated with the success of community case management of malaria? A qualitative study from Burkina Faso. Soc Sci Med 2015, 124: 232-240.

24