Criteria for selecting implementation frameworks and theories among - PowerPoint PPT Presentation

1 Criteria for selecting implementation frameworks and theories among implementation researchers and practitioners Sarah A. Birken, PhD The University of North Carolina at Chapel Hill @birkensarah

2 2 @birkensarah

3 How do I choose a theory? 3 @birkensarah

Criteria for selecting theories and frameworks 4 • Snowball approach beginning with seminal articles 1-3 in implementation science regarding framework and theory selection • Iterative refinement based on discussion among authors 1. Holmström, J, Truex D: What does it mean to be an informed IS researcher? Some criteria for the selection and use of social theories in IS research. Information Systems Research Seminar in Scandinavia (IRIS) 2001, 313 – 326. 2. The Improved Clinical Effectiveness through Behavioral Research Group (ICEBeRG): Designing theoretically informed implementation interventions. Implementation Science 2006, 1:4. 3. Wacker JG: A definition of theory: research guidelines for different theory-building research methods in operations management. Journal of Operations Management 1998, 16: 361-385. 4 @birkensarah

Criteria for selecting theories and frameworks 5 1. Process guidance: Provision of a step-by-step approach for application 2. Inclusion of change strategies/techniques: Provision of specific method(s) for promoting change in implementation-related processes and/or outcomes • Falsifiability : Verifiable; ability to be supported with empirical data 3. Associated research method (e.g., informs qualitative interviews; associated with a validated questionnaire or methodology for constructing one): Recommended or implied method to be used in an empirical study that uses the framework or theory • Simplicity/parsimony: Relatively few assumptions are used to 4. Uniqueness: Ability to be distinguished from other theories or frameworks 5. Falsifiability: Verifiable; ability to be supported with empirical data explain effects 6. Inclusion of a diagrammatic representation: Elaboration in a clear and useful figure representing the concepts within and their inter-relations (if applicable) • Analytic level (e.g., individual, organization, system): Level of the 7. Application to a specific setting (e.g., hospitals, schools)/population (e.g., cancer): Intentional or historical use in studies related to particular groups and/or conditions 8. Specificity of causal relationships among constructs: Summary, explanation, organization, and description of relationships among constructs socioecological system at which constructs in the framework or 9. Empirical support: Use in empirical studies with results relevant to the framework or theory, contributing to cumulative theory-building 10. Disciplinary origins: Philosophical foundations theory lie 11. Simplicity/parsimony: Relatively few assumptions are used to explain effects 12. Fecundity: Offers a rich source for generating hypotheses • Generalizability : Applicability to various disciplines, settings, and 13. Analytic level (e.g., individual, organization, system): Level of the socioecological system at which constructs in the framework or theory lie 14. Generalizability: Applicability to various disciplines, settings, and populations populations 15. Logical consistency/plausibility: Inclusion of meaningful, face-valid explanations of proposed relationships 16. Explanatory power/testability: Ability to provide explanations around variables and effects; generates hypotheses that can be empirically tested 17. Outcome of interest: Conceptual centrality of the variable to which included constructs are thought to be related 18. Disciplinary approval: Frequency of use, popularity, acceptability, and perceptions of influence among a given group of scholars or reviewers, country, funding agencies, etc; endorsement or recommendation by credible authorities in the field 19. Description of a change process: Provides an explanation of how changes in process factors lead to changes in implementation-related outcomes 5 @birkensarah

6 6 @birkensarah

7 7 @birkensarah

Background 8 • Implementation frameworks and theories guide evidence translation, identify implementation determinants, and evaluate implementation. • The lack of guidance for selecting frameworks and theories may limit their (appropriate) use. 8 @birkensarah

Background 9 • Evidence exists of superficial use of frameworks and theories (e.g., Kirk et al., 2016; Eckstein & El Zarrad , 2012)… • This may be evidence of lack of clarity around selecting appropriate frameworks and theories. 9 @birkensarah

Background 10 • There is a need to promote appropriate selection and use of frameworks and theories. • Diminishing silos across disciplines, countries, degree types, and institution types in the following may promote appropriate selection and use of frameworks and theories. 10 @birkensarah

Objective 11 • Understand implementation researchers’ and practitioners’ criteria for selecting frameworks and theories 11 @birkensarah

12 Variables 12 @birkensarah

Respondent demographics 13 • Highest degree earned • Seniority (years of experience, external funding, articles published, etc.) • Institution type (e.g., academic, CRO) • Location of institution (country) 13 @birkensarah

Research vs. practice 14 Which of the following best describes the nature of your work? Please select one.  I conduct or collaborate on implementation research studies  proceed  I implement programs and/or engage in quality improvement initiatives  proceed  I do some of both  proceed  None of the above  do NOT proceed 14 @birkensarah

Criteria for selecting theories and frameworks 15 Thinking of your research in the last two years, which of the following characteristics did you consider when selecting a framework or theory for your research? Please select all that apply.  Process guidance : Provision of a step-by-step approach for application  Inclusion of change strategies/techniques : Provision of specific method(s) for promoting change in implementation-related processes and/or outcomes  Associated research method (e.g., informs qualitative interviews; associated with a validated questionnaire or methodology for constructing one): Recommended or implied method to be used in an empirical study that uses the framework or theory  Uniqueness: Ability to be distinguished from other theories or frameworks  Falsifiability: Verifiable; ability to be supported with empirical data  Inclusion of a diagrammatic representation: Elaboration in a clear and useful figure representing the concepts within and their inter-relations (if applicable)  Application to a specific setting (e.g., hospitals, schools)/population (e.g., cancer): Intentional or historical use in studies related to particular groups and/or conditions  Specificity of causal relationships among constructs: Summary, explanation, organization, and description of relationships among constructs  Empirical support: Use in empirical studies with results relevant to the framework or theory, contributing to cumulative theory-building  Disciplinary origins: Philosophical foundations  Simplicity/parsimony: Relatively few assumptions are used to explain effects  Fecundity: Offers a rich source for generating hypotheses  Analytic level (e.g., individual, organization, system): Level of the socioecological system at which constructs in the framework or theory lie  Generalizability: Applicability to various disciplines, settings, and populations  Logical consistency/plausibility: Inclusion of meaningful, face-valid explanations of proposed relationships  Explanatory power/testability: Ability to provide explanations around variables and effects; generates hypotheses that can be empirically tested  Outcome of interest: Conceptual centrality of the variable to which included constructs are thought to be related  Disciplinary approval: Frequency of use, popularity, acceptability, and perceptions of influence among a given group of scholars or reviewers, country, funding agencies, etc; endorsement or recommendation by credible authorities in the field  Description of a change process: Provides an explanation of how changes in process factors lead to changes in implementation-related outcomes  None of the above 15

Additional criteria 16 In the space provided, please list any characteristics NOT included in the list above that you considered when selecting a framework or theory for your research or practice in the past two years. 16 @birkensarah

Criteria ranking 17 In the space provided, please list the three most important criteria in order of importance Most important Second most important Third most important 17 @birkensarah

Analysis 18 • Descriptive statistics 18 @birkensarah

19 Results 19 @birkensarah

20 Study sample (n = 224) 20 @birkensarah

Highest degree earned 21 N=186 21 @birkensarah

Seniority 22 Item N Mean SD Minimum Maximum Range 184 13.8 8.9 1 54 53 Years of research Years of implementation research 184 7.4 7.1 0.5 44 43.5 Number of implementation research 179 6.0 7.8 0 50 50 projects Number of articles published 180 36.6 61.4 0 425 425 Number of articles in implementation 180 10.2 18.7 0 150 150 science published 184 No = 35.9% Yes = 63.0% PI of externally funded study? 22 @birkensarah