Criteria for selecting implementation frameworks and theories among - - PowerPoint PPT Presentation

1

Criteria for selecting implementation frameworks and theories among implementation researchers and practitioners

Sarah A. Birken, PhD The University of North Carolina at Chapel Hill @birkensarah

2

2

@birkensarah

3

How do I choose a theory?

3

@birkensarah

4

• Snowball approach beginning with seminal articles1-3 in

implementation science regarding framework and theory selection

• Iterative refinement based on discussion among authors

4 1. Holmström, J, Truex D: What does it mean to be an informed IS researcher? Some criteria for the selection and use of social theories in IS research. Information Systems Research Seminar in Scandinavia (IRIS) 2001, 313–326. 2. The Improved Clinical Effectiveness through Behavioral Research Group (ICEBeRG): Designing theoretically informed implementation interventions. Implementation Science 2006, 1:4. 3. Wacker JG: A definition of theory: research guidelines for different theory-building research methods in operations management. Journal of Operations Management 1998, 16: 361-385.

Criteria for selecting theories and frameworks

@birkensarah

5

1. Process guidance: Provision of a step-by-step approach for application 2. Inclusion of change strategies/techniques: Provision of specific method(s) for promoting change in implementation-related processes and/or

• utcomes

3. Associated research method (e.g., informs qualitative interviews; associated with a validated questionnaire or methodology for constructing one): Recommended or implied method to be used in an empirical study that uses the framework or theory 4. Uniqueness: Ability to be distinguished from other theories or frameworks 5. Falsifiability: Verifiable; ability to be supported with empirical data 6. Inclusion of a diagrammatic representation: Elaboration in a clear and useful figure representing the concepts within and their inter-relations (if applicable) 7. Application to a specific setting (e.g., hospitals, schools)/population (e.g., cancer): Intentional or historical use in studies related to particular groups and/or conditions 8. Specificity of causal relationships among constructs: Summary, explanation, organization, and description of relationships among constructs 9. Empirical support: Use in empirical studies with results relevant to the framework or theory, contributing to cumulative theory-building

• 10. Disciplinary origins: Philosophical foundations
• 11. Simplicity/parsimony: Relatively few assumptions are used to explain effects
• 12. Fecundity: Offers a rich source for generating hypotheses
• 13. Analytic level (e.g., individual, organization, system): Level of the socioecological system at which constructs in the framework or theory lie
• 14. Generalizability: Applicability to various disciplines, settings, and populations
• 15. Logical consistency/plausibility: Inclusion of meaningful, face-valid explanations of proposed relationships
• 16. Explanatory power/testability: Ability to provide explanations around variables and effects; generates hypotheses that can be empirically tested
• 17. Outcome of interest: Conceptual centrality of the variable to which included constructs are thought to be related
• 18. Disciplinary approval: Frequency of use, popularity, acceptability, and perceptions of influence among a given group of scholars or reviewers,

country, funding agencies, etc; endorsement or recommendation by credible authorities in the field

• 19. Description of a change process: Provides an explanation of how changes in process factors lead to changes in implementation-related outcomes

5

• Falsifiability: Verifiable; ability to be supported with empirical data
• Simplicity/parsimony: Relatively few assumptions are used to

explain effects

• Analytic level (e.g., individual, organization, system): Level of the

socioecological system at which constructs in the framework or theory lie

• Generalizability: Applicability to various disciplines, settings, and

populations

Criteria for selecting theories and frameworks

@birkensarah

6

6

@birkensarah

7

7

@birkensarah

8

Background

• Implementation frameworks and theories guide evidence

translation, identify implementation determinants, and evaluate implementation.

• The lack of guidance for selecting frameworks and theories

may limit their (appropriate) use.

8

@birkensarah

9

Background

• Evidence exists of superficial use of frameworks and theories

(e.g., Kirk et al., 2016; Eckstein & El Zarrad, 2012)…

• This may be evidence of lack of clarity around selecting

appropriate frameworks and theories.

9

@birkensarah

10

Background

• There is a need to promote appropriate selection and use of

frameworks and theories.

• Diminishing silos across disciplines, countries, degree types,

and institution types in the following may promote appropriate selection and use of frameworks and theories.

10

@birkensarah

11

Objective

• Understand implementation researchers’ and practitioners’ criteria

for selecting frameworks and theories

11

@birkensarah

12

Variables

12

@birkensarah

13

Respondent demographics

• Highest degree earned
• Seniority (years of experience, external funding, articles

published, etc.)

• Institution type (e.g., academic, CRO)
• Location of institution (country)

13

@birkensarah

14

Which of the following best describes the nature of your work? Please select one.  I conduct or collaborate on implementation research studies  proceed  I implement programs and/or engage in quality improvement initiatives  proceed  I do some of both  proceed  None of the above  do NOT proceed

Research vs. practice

14

@birkensarah

15

Criteria for selecting theories and frameworks

15

Thinking of your research in the last two years, which of the following characteristics did you consider when selecting a framework or theory for your research? Please select all that apply.

 Process guidance: Provision of a step-by-step approach for application  Inclusion of change strategies/techniques: Provision of specific method(s) for promoting change in implementation-related processes and/or outcomes  Associated research method (e.g., informs qualitative interviews; associated with a validated questionnaire or methodology for constructing one): Recommended or implied method to be used in an empirical study that uses the framework or theory  Uniqueness: Ability to be distinguished from other theories or frameworks  Falsifiability: Verifiable; ability to be supported with empirical data  Inclusion of a diagrammatic representation: Elaboration in a clear and useful figure representing the concepts within and their inter-relations (if applicable)  Application to a specific setting (e.g., hospitals, schools)/population (e.g., cancer): Intentional or historical use in studies related to particular groups and/or conditions  Specificity of causal relationships among constructs: Summary, explanation, organization, and description of relationships among constructs  Empirical support: Use in empirical studies with results relevant to the framework or theory, contributing to cumulative theory-building  Disciplinary origins: Philosophical foundations  Simplicity/parsimony: Relatively few assumptions are used to explain effects  Fecundity: Offers a rich source for generating hypotheses  Analytic level (e.g., individual, organization, system): Level of the socioecological system at which constructs in the framework or theory lie  Generalizability: Applicability to various disciplines, settings, and populations  Logical consistency/plausibility: Inclusion of meaningful, face-valid explanations of proposed relationships  Explanatory power/testability: Ability to provide explanations around variables and effects; generates hypotheses that can be empirically tested  Outcome of interest: Conceptual centrality of the variable to which included constructs are thought to be related  Disciplinary approval: Frequency of use, popularity, acceptability, and perceptions of influence among a given group of scholars or reviewers, country, funding agencies, etc; endorsement or recommendation by credible authorities in the field  Description of a change process: Provides an explanation of how changes in process factors lead to changes in implementation-related outcomes  None of the above

16

Additional criteria

In the space provided, please list any characteristics NOT included in the list above that you considered when selecting a framework or theory for your research or practice in the past two years.

16

@birkensarah

17

Criteria ranking

17

In the space provided, please list the three most important criteria in order of importance Most important Second most important Third most important @birkensarah

18

Analysis

• Descriptive statistics

18

@birkensarah

19

Results

19

@birkensarah

20

Study sample (n = 224)

20

@birkensarah

21

Highest degree earned

21

N=186

@birkensarah

22

Seniority

Item N Mean SD Minimum Maximum Range

Years of research

184 13.8 8.9 1 54 53

Years of implementation research

184 7.4 7.1 0.5 44 43.5

Number of implementation research projects

179 6.0 7.8 50 50

Number of articles published

180 36.6 61.4 425 425

Number of articles in implementation science published

180 10.2 18.7 150 150

PI of externally funded study?

184 No = 35.9% Yes = 63.0%

22

@birkensarah

23

Institution type

23

10 20 30 40 50 60 70 80 Academic Hospital-based research institute Government Service provider Industry Other

Percent of Respondents N=182 Type of Institution

@birkensarah

24

Location of institution

24

@birkensarah

25

Research vs. practice

25

Researchers 42% Practitioners 11%

47%

N=223

@birkensarah

26

Response % Analytic level 58 Logical consistency/plausibility 56 Empirical support 53 Description of a change process 54 Generalizability 47 Outcome of interest 41 Inclusion of a diagrammatic representation 41 Inclusion of change strategies/techniques 44 Application to a specific setting (e.g., hospitals, schools)/population (e.g., cancer) 44 Disciplinary approval 34 Associated research method 40 Process guidance 39 Specificity of causal relationships among constructs 32 Explanatory power/testability 33 Simplicity/parsimony 32 Falsifiability 15 Disciplinary origins 18 Fecundity 10 Uniqueness 13 26

@birkensarah

27

Number of Criteria Used

27

N=212

@birkensarah

28

Most important criteria

Criterion Most Important (%)

Empirical support 17 Explanatory power/testability 13 Applicability to setting 10 Description of change process 9 Analytic level 6

28

N=176

@birkensarah

29

Additional criteria listed by respondents

• Familiarity: Extent to which PI or research team is familiar with the

theory or framework

• Degree of specificity: Extent to which included constructs are

comprehensive of implementation determinants or specific to a particular set of implementation determinants

• Accessibility: Extent to which non-experts are able to understand,

apply, and operationalize framework or theory’s propositions

29

@birkensarah

30

Pragmatic vs. political criteria

• Pragmatic: contributes substantively to the conceptualization of

the research question (e.g., generalizability)

• 42% report using at least one political criterion (i.e. disciplinary

approval or uniqueness) when selecting theories/frameworks

30

“My advisor told me to.”

@birkensarah

31

31

“It still mostly feels like a leap in the dark which plays to what others (e.g. funders) are likely to recognize/value rather than a more informed judgement about which would be the most appropriate framework or theory for THIS project.” “To some degree selection is arbitrary. There are probably several models that would be fruitful, and I tend to use

• nes that are familiar to me.”

“As someone who implements programs we often have to rely on external experts, academics or researchers to find relevant frameworks or suggest theories/frameworks that will fit our particular needs. If there was a way of making [theories/frameworks] more accessible outside of academic circles, they may be more widely used.”

@birkensarah

32

Discussion

• Implementation researchers and practitioners use an unwieldy

number of criteria for selecting frameworks and theories.

• Criteria used for selection are more idiosyncratic than systematic.
• Implementation researchers and practitioners may benefit from a

tool that guides framework and theory selection (i.e., a checklist).

32

@birkensarah

33

Guidance for selecting theories and frameworks

33

Aim 1 Aim 2 Aim 3

Identify a comprehensive set of criteria for selecting frameworks and

• theories. In a concept mapping exercise, implementation researchers and

practitioners will identify and rate conceptually distinct categories of criteria that were included in preliminary studies. Develop a user-friendly tool to guide framework and theory selection. We will use published guidelines for enhancing visual communication and properties to optimize the design of a tool that features the criteria identified in Aim 1. Assess the utility of the tool from the perspective of implementation researchers and practitioners. We will use cognitive interviews to solicit feedback on the tool.

@birkensarah

34

Future research

• A conference intended to promote framework and theory use

among implementation researchers and practitioners

34

@birkensarah

35

Future research

• Guidelines for applying theory
• Literature review to identify existing guidelines
• Expert consultation
• Examples of framework and theory application in high-quality grants

and articles

35

How do I apply theory?

@birkensarah

36

Acknowledgements

• Consortium for Implementation Science
• Catherine Rohweder
• Jen Scott
• Chris Shea
• Emily Haines
• Alessandra Bassalobre Garcia
• North Carolina Translational & Clinical Sciences Institute:

SOTSR11601

36

@birkensarah

37

Questions?

37

@birkensarah