Evidence-Based Practice: Myths and Realities Bruce A. Thyer, Ph.D., - - PowerPoint PPT Presentation

1

Evidence-Based Practice: Myths and Realities

Bruce A. Thyer, Ph.D., LCSW, BCBA-D College of Social Work, Florida State University

A Keynote Address Presented at the Michigan Autism Conference, 16 September 2016 Kalamazoo, MI

2

What is Evidence Based Practice?

• “Evidence-based practice requires the

integration of the best research evidence with

• ur clinical expertise and our patient’s unique

values and circumstances”

From Strauss et al. (2011). Evidence-based medicine: How to practice and teach EBM (fourth edition, p. 1). New York: Elsevier.

3

Note the equivalent importance of ALL these factors in the EBP process

• *

4

What is ‘Best Research Evidence’?

• Clinically relevant research from basic and

applied scientific investigations, especially drawing from intervention research evaluating the outcomes of social work services, and from studies on the reliability and validity of assessment measures.

5

Higher End of Internal Validity (in terms of causal inference)

• Systematic Reviews (highest form of evidence)
• Meta-analyses
• Multi-site Randomized Clinical Trials
• Individual RCTs
• Quasi-experiments
• Pre-experiments
• Single Subject Studies
• Correlational Studies/Epidemiological Studies
• Qualitative Research
• Narrative Case Studies
• Basic Science Studies
• Expert or consensus opinion, Theory (lowest form
• f evidence)

Lower End of Internal Validity

6

‘Best Evidence’ Means Best Available

• Look for relevant systematic reviews, then meta-

analyses, then RCTs, then quasi-experiments, etc. Integrate this best available evidence into your decision-making practice. EBP does NOT depend on having a large body of RCT’s available to consult. It does depend on one examining the best available evidence.

• There is ALWAYS evidence, even if it is of low

quality.

7

What are Client Values?

• The unique preferences, concerns and

expectations each client brings to a clinical encounter with a social worker, and which must be integrated into practice decisions if they are to serve the client.

• A thorough consideration of ethical

considerations and client considerations is integral to the EBP model.

8

What is Clinical Expertise?

• Our ability to use our education, interpersonal skills

and past experience to assess client functioning, diagnose mental disorders and/or other relevant conditions, including environmental factors, and to understand client values and preferences.

• Clinical expertise factors, costs, available resources,
• etc. are integral to the EBP model.
• Research findings are NOT accorded greater weight.

All are compellingly important.

Many Interventions Might Have Strong Research Support but Not be Acceptable

• Client may not wish it
• May be religiously objectionable

(e.g., Jehovahs’ Witnesses may decline blood, or a Muslim may decline a pig valve implant)

• May be illegal or unethical
• Costs too much
• So a lesser supported intervention may be provided, and still be

consistent with EBP!

9

10

What are the Major Steps of Evidence-based Practice?

1. Convert the need for information into an answerable questions(s).

• 2. Track down the best available evidence to answer each

question. 3. Critically evaluate this evidence in terms of its validity, impact, and potential relevance to our client. 4. Integrate relevant evidence with our own clinical expertise and client values and circumstances. 5. Evaluate our expertise in conducting Steps 1-4 above, and evaluate the outcomes of our services to the client, especially focusing on an assessment of enhanced client functioning and/or problem resolution.

11

What are ‘Answerable Questions’?

• 2. A question including some aspect of the client’s behavior or
• condition. As in
• What psychosocial interventions reduce self-injurious behavior?
• What educational programs are the most successful in teaching

children to read?

• How can schools reduce the social isolation of students with

autism?

• What treatments are effective in improving social skills of youth

with autism?

• 1. A question with a verb, as in
• What has been shown to help….? Or
• What psychosocial treatments work….?
• What community-based interventions reduce….?
• What group therapies improve….?

12

13

How Can You Track Down the Best Available Evidence?

There are LOTS of resources!

• Practice-research journals, as in

– Journal of Applied Behavior Analysis – Behavior Analysis in Practice – Behavior Analysis: Research and Practice – Journal of Autism and Developmental Disorders – Focus on Autism and other Developmental Disabilities

14

Science-based Textbooks, as in

• Understanding Applied Behavior Analysis, Second Edition:

An Introduction to ABA for Parents, Teachers, and other Professionals 2nd Edition

• Applying Behavior Analysis Across the Autism Spectrum
• The Parent's Guide to In-Home ABA Programs: Frequently

Asked Questions about Applied Behavior Analysis for your Child with Autism

• Applied Behavior Analysis for Children with Autism

Spectrum Disorders

15

• Science-based Websites, as in:
• Autism Speaks - https://www.autismspeaks.org/
• National Autism Center at the May Institute

http://www.nationalautismcenter.org/about-nac/mission/

• Autism Consortium -

http://www.autismconsortium.org/families/understanding- autism-and-treatments/

• The ABAI Learning Center -

https://www.abainternational.org/learning-center.aspx

• BE CAUTIOUS – There are many pseudoscientific websites promoting

none-science-based treatments, such as this one promoting homeopathy as a treatment for autism: http://www.drhomeo.com/autism/autism-and-

homeopathy-a-miraculus- cure/?gclid=CMyXsoXPiM8CFUM2gQodg5oNig

16

Science-based Professional Conferences, such as:

• Association for Behavior Analysis International (ABAI)
• Various state-chapter conferences of ABAI
• Association of Professional Behavior Analysts
• National Autism Conference
• Autism Society National Conference

Caveat – not all presentations will necessarily be science-based.

17

3. How Can You Critically Evaluate the Available Evidence?

• Develop critical appraisal skills in evaluating research yourself. (a

bottom-up search)

• Seek out and rely on credible groups which have already done

this (e.g. ABAI, Cochrane and Campbell Collaboration, APA’s Division 12’s lists of ESTs, SAMSHA, California Clearing House…etc.) (a top-down search)

18

Pros and Cons of Each Approach – Search Yourself

• Searching yourself brings you into contact with the most current

research.

• Searching yourself can be difficult to do in a comprehensive

manner.

• Many stakeholders such as parents, may lack the scientific critical

thinking skills needed to sort out legitimate research studies from poorly conducted one.

• Many so-called “Systematic Reviews” are not of high quality

and the term is promiscuously applied. Some interventions are designed as an ‘evidence-based practice’ for autism, when they are really not. The Cochrane and Campbell Collaborations provide access to the highest quality SRs.

19

Pros and Cons of Each Approach – Rely on Summaries

• f Research Prepared by Others
• Locating a high quality systematic review dealing with a question

important to you and your client is a tremendous time-saver. Others have done all the hard work for you!

• Many so-called “Systematic Reviews” are not of high quality and

the term is promiscuously applied. The Cochrane and Campbell Collaborations provide access to the highest quality SRs.

• Getting access to the Cochrane and Campbell Libraries may be

difficult for stakeholders not located in universities.

• There may not be a systematic review relevant to your

informational needs.

20

High Quality Systematic Reviews are:

• Systematic, in that a replicable search strategy to locate relevant

literature is clearly described and adhered to.

• As unbiased as possible. Interrater reliability checks are used to

evaluate potential papers.

• Draw upon the international literature, not just English language

journals and books and conference.

• Make use of interdisciplinary teams of researchers, and non-

professional stakeholders. These teams are often international in scope.

• Follow previously published SR protocols.
• Regularly updated.

21

The Cochrane Collaboration has 23 systematic reviews dealing with the topic of AUTISM, including:

• Early Intensive Behavioral Intervention (EIBI) for Increasing

Functional behaviors and Skills in Young People with ASD.

• Early Intervention Delivered by Parents for Young Children

with ASD

• Social Skills Groups for People Aged 6-21 with ASD
• Auditory Integration Therapy for ASD
• Intravenous Secretin for ASDAcupuncture for People with ASD
• Chelation for ASD
• Gluten and Casein-Free Diets for ASD
• Omega-3 Fatty Acids for ASD
• Using the Combined Vaccine for Protection of Children Against

Measles, Mumps and Rubella

22

23

The Campbell Collaboration has a limited number of completed systematic reviews related to Autism. More are in preparation:

• Effectiveness of Adult Employment Assistance for

Persons with ASD

• Pre-graduation Transition Services for Persons with

ASD: Effects on Employment Outcomes

• Electronic Assistive Technology for Improving Social

and Behavioral Outcomes for Individuals with ASD

• Interventions for Anxiety in School-aged Children with

ASD

24

Systematic Reviews Published by the Cochrane and Campbell Collaborations are the most credible source of information.

• Less credible are SRs published in journals and on

websites, as these lack the meticulous quality assurance

• versight associated with Cochrane and Campbell.
• Another common source of information are websites

that purport to list and describe research-supported interventions and so-called evidence-based practices. Some are promoted by the federal and state government, and other by private groups.

25

Points of Congruence between EBP and Behavior Analysis

• 1. Interestingly, EBP is VERY encouraging of

practitioners using N = 1 experiments to evaluate the effects of treatment with individual clients!

All editions of Evidence-based Medicine have included a small chapter on using N = 1 experiments, primarily the randomized alternating treatments design. But there seems little awareness of other experimental N = 1 studies, like the reversal or multiple baseline designs.

One major reference on EBP claims that N =1 Randomized Controlled Trials represents the HIGHEST standard of evidence of treatment effectiveness! Solid evidence that the intervention worked with YOUR Client!

A Hierarchy of Strength of Evidence for Treatment Decisions* 1. N of 1 randomized controlled trial (highest) 2. Systematic reviews of randomized controlled trials 3. Single RCT 4. Systematic review of observational outcome studies 5. Single observational outcome study 6. Physiologic Studies 7. Unsystematic clinical observations (low) *G. Guyatt & D. Rennie (Eds.) (2002). Users’ Guides to the Medical Literature: Essential of Evidence-based Clinical

• Practice. Chicago: American Medical Association. p. 12.

Recall the 5th step of the EBP Decision-Making Approach*

5. Evaluate our expertise in conducting Steps 1-4 above, and evaluate the outcomes of our services to the client, especially focusing on an assessment of enhanced client functioning and/or problem resolution.

An Example of a N = 1 Trial Taken from the CONSORT Website

http://www.consort-statement.org/extensions?ContentWidgetId=47627

31

Professional and Ethical Compliance Code for Behavior Analysts*

1.01 Reliance on Scientific Knowledge. Behavior analysts rely on professionally derived knowledge based on science and behavior analysis when making scientific

• r professional judgments in human service provision, or when engaging in scholarly or

professional endeavors. 2.09 Treatment/Intervention Efficacy. (a) Clients have a right to effective treatment (i.e., based on the research literature and adapted to the individual client). Behavior analysts always have the obligation to advocate for and educate the client about scientifically supported, most-effective treatment procedures. Effective treatment procedures have been validated as having both long-term and short-term benefits to clients and society. (c) In those instances where more than one scientifically supported treatment has been established, additional factors may be considered in selecting interventions, including, but not limited to, efficiency and cost-effectiveness, risks and side-effects of the interventions, client preference, and practitioner experience and training. *http://bacb.com/wp-content/uploads/2016/03/160321-compliance-code-english.pdf

32

Recall the Second Step in the Five-Step EBP Decision- Making Model

2. “Critically evaluate this evidence in terms of its validity, impact, and potential relevance to our client”. And the very definition of EBP “Evidence-based practice requires the integration of the best research evidence with

• ur clinical expertise and our patient’s unique values and circumstances” (p. 1)

Patient values = “the unique preferences, concerns and expectation each patient brings to a clinical encounter and must be integrated into clinical decisions is they are to serve the patient. (p. 1)

From Strauss et al. (2011). Evidence-based medicine: How to practice and teach EBM (fourth edition, p. 1). New York: Elsevier.

33

34

See, for example,

• “ In addition to clinical expertise, the clinician

requires compassion, sensitive listening skills, and broad perspectives from the humanities and social

• sciences. These attributes allow understanding of

patient’s illnesses in the context of their experience, personalities and cultures…For some of the patients and problems, this discussion should involve the patient’s family.”

cited from Guyatt, G. & Rennie, D. (Eds.) (2002). Users’ guides to the medical literature: Essentials of evidence-based clinical practice (p. 15). Chicago, IL: American Medical Association.

Compare these EBP principles with the ethical guidelines for the practice of behavior analysis -

4.02 Involving Clients in Planning and Consent. Behavior analysts involve the client in the planning of and consent for behavior-change programs. 4.03 Individualized Behavior-Change Programs. (a) Behavior analysts must tailor behavior-change programs to the unique behaviors, environmental variables, assessment results, and goals of each client. c.f. Professional and Ethical Compliance Code for Behavior Analysts http://bacb.com/wp-content/uploads/2016/03/160321-compliance-code-english.pdf

35

.

Shared Points of Congruence Between EBP and ABA?

• 1. Both base assessment methods on

scientific evidence!

• 1. Both evaluate treatment outcome with single-

subject experimental evaluations

• 2. Both assert that client preferences must be taken

into account when making treatment decisions.

37

Unfortunately, many people confuse the five-step decision-making process of EBP with selecting empirically-supported

• treatments. There are subtle but important

differences in these two approaches.

38

What are Empirically-Supported Treatments and Where Do They Come From?

Division 12 (Clinical Psychology of the APA)

• rganized a “Task for on Promotion and

Dissemination of Psychological Procedures” in the early 1990s. Its purpose was to “publish information for both the practitioner and the general public on the random assignment, controlled outcome study literature of psychotherapy and of psychoactive medications.”

39

The Task Force had Two Sequential Tasks:

• 1. To develop evidentiary standards to be used

to designate a given treatment/assessment methods as “empirically validated” (later changed to “empirically supported”. 2. To review the literature and publish lists of treatments that met or did not meet these evidentiary standards.

40

What Evidentiary Standards Did They Develop? They (APA, Division 12) came up with two sets of standards or evidence benchmarks, one to designate an treatment as ‘empirically supported”(hence ESTs) or well supported, and another, less stringent

• ne, used to designate an intervention as promising
• rprobably efficacious.

APA’s Division 12, Section 3

• Definitions of Empirically Supported Treatments

Best Support (“Well-Established Treatments”)

• I. At least two good between group design experiments

demonstrating efficacy in one or more of the following ways:

• a. Superior to pill placebo, psychological placebo, or another

treatment.

• b. Equivalent to an already established treatment in experiments

with adequate statistical power (about 30 per group; cf. Kazdin & Bass, 1989) OR

APA’s Division 12, Section 3

OR

• II. A large series of single case design experiments

(n > 9) demonstrating efficacy. These experiments must have:

• a. Used good experimental designs
• b. Compared the intervention to another treatment as

in I.a.

• And (for both pathways)

APA’s Division 12, Section 3

AND

• Further criteria for both I and II:
• III. Experiments must be conducted with

treatment manuals.

• IV. Characteristics of the client samples

must be clearly specified.

• V. Effects must have been demonstrated

by at least two different investigators or teams of investigators.

44

OK – Where are these lists of ‘so-called empirically supported treatments be found? Several publication pathways emerged from the Task Force’s efforts: Initially, one book - Nathan, P. E. & Gorman, J. M. (Eds.) (2007). A Guide to Treatments That Work (third edition). New York: Oxford University Press

45

And a series of articles and special issues of journals

These older articles are available for free at: http://www.psych.upenn.edu/~dchamb/ESTs/EST _2011.html You can also find their current lists of ESTs on this website, broken down by “Treatments” and by “Disorders” (this list is focused on so- called mental disorders only). See: https://www.div12.org/psychological-treatments/

46

47

48

49

50

51

52

53

54

55

HOWEVER!

There are NO entries for Autism on this Division 12 website listing Research-Supported Psychological Treatments!

Autism-serving organizations could take the initiative in promoting the inclusion of science-based therapies in this website.

56

Problems with the Psychologists’ EST Initiative for Behavior Analysis?

• Few Behavior Analysts on Committees.
• Strong Bias to Preferring Between Groups

Designs (and their associated limitations)

• Very few Contemporary Behavior Analytic

Treatments Made the Lists. None, as far as I can tell, via the route of a series of successful single-case experimental designs.

REMEMBER – the evidentiary standards DO PERMIT inclusion of treatments supported via experimental within-subject studies (N = 1). Behavior analysts and other Autism Stakeholders could take the national lead on collaborating with Division 12 to see that ABA interventions are not overlooked. All it takes are 9+ N = 1 experimental studies!

Problems with the EST Initiative for Behavior Analysis?

The book, A Guide to Treatments that Work, contains little about behavior analysis, although behavior therapy and CBT are well represented, reflecting the Editors’ bias towards favoring evidence derived from between-group studies.

60

See also:

A new Division 12 developed, edited and supported book series titled “Keeping up with the Advances in Psychotherapy: Evidence-based Practice”, published by Hogrefe & Huber. Note the crucial terminology change from ‘empirically supported to “evidence-based” This is a problem. These are different things.

61

While the EST movement remains alive and well (under its new name of Research Supported Treatments), it has largely been overtaken by the Evidence-based Practice Movement

62

WARNING!

Folks who wish to intelligently discuss evidence-based practice should be very familiar with the primary source readings on EBP. It is NOT the SAME as Empirically Supported Treatments!

63

EBP is a PROCESS of learning, it is NOT A LISTING OF EFFECTIVE TREATMENTS!

Crucial Definitional Terms such as

• “Best Research Evidence”,
• “Clinical Expertise”
• “Patient Values” and
• “Patient Circumstances”

are all operationalized reasonably well.

What Should Autism Stakeholders, Including Behavior Analysts, Do and NOT Do?

When we talk about interventions that are supported by credible research, please use the language of research-supported treatments, or interventions, call these RSIs. When we are talking about evidence-based practice, lets keep in mind that this is a 5-step-decision- making process, not a listing of interventions.

65

There is no such thing as EVIDENCE-BASED PRACTICES

It is mixing apples and oranges to refer to evidence-based practices, when we really mean research supported treatments! See

Thyer & Pignotti (2011). Evidence-based practices do not exist. Clinical Social Work Journal, 38, 328-333.

66

In fact, nowhere in the Campbell or Cochrane Collaborations do you see lists of endorsed treatments. Such lists would actually be antithetical to EBP, since these ignore clinical variables, ethics, and clinical expertise, other elements valued equally with scientific support.

67

The EST program is less scientifically and professionally credible than EBP. When we talk about EBP in terms only

• f lists of approved therapies, we tar

EBP with the deficiencies of the EST model, distorting EBP.

68

Some Problems with lists of ESTs

• “One legitimate criticism is that the EST list is based on an
• verly simple “all or none” model of effectiveness: A

treatment is either empirically supported or it is not. Yet the true state of affairs is likely far more more complex” (ABCT website, on ESTs)

• Such lists of ESTs ignore ethical considerations, client

preferences, resource consideration and the adequacy or clinical expertise.

• They also focus on positive studies and ignore negative
• utcome studies. (a treatment with two positive studies and 8

negative ones could be considered empirically supported!)

• They are based on p-values in determining effectiveness and

ignore effect sizes of treatments.

69

See, for example,

• “As a distinctive approach to patient care,

EBM involved two fundamental principles. First, evidence alone is never sufficient to make a clinical decision. Decision makers must always trade the benefits and risk, inconvenience and costs associated with alternative management strategies and in doing so consider the patient’s values.” (emphasis added)

cited from Guyatt, G. & Rennie, D. (Eds.) (2002). Users’ guides to the medical literature: Essentials of evidence-based clinical practice (p. 8). Chicago, IL: American Medical Association.

70

Also, the EBP Process does NOT involve recourse to Practice

• Guidelines. Some Problems with Practice Guidelines include
• They are usually created by members of one discipline, and

fail to adequately take into account interdisciplinary literature.

• Disciplinary prejudices are rife (PGs prepared by psychiatrists

tend to ignore effective psychosocial treatments)

• ‘Expert consensus’ sometimes overrules scientific

considerations.

• They are usually not too comprehensive, and ignore the ‘gray’

literature.

See criticisms of practice guidelines in Straus, S. E. et al. (2011). Evidence-based medicine: How to practice and teach it (pp. 128-129). New York: Churchill Livingston.

• r Thyer, B. A. (2003). Social work should help develop interdisciplinary evidence-based

practice guidelines, not discipline-specific ones. In A. Rosen & E. K. Proctor (Eds.). Developing practice guidelines for social work interventions (pp. 128-139). New York; Columbia University Press.

71

Other Misconceptions

“There is an inadequate foundation of high quality evidence regarding the problem of XXX. Therefore, we cannot be expected to make use of the EBP model.” EBP does not require the existence of lots of high quality evidence. It does require the practitioner to seek out, appraise, and judge the applicability of the highest quality available evidence. There is always evidence, even it is consists of informed clinical opinion, or theoretical systems.

72

Summary

It is possible that the EBP process model represents a significant positive step in the professional maturation of human services and in our ability to genuinely help persons with autism and their families, and to implement effective social policies and programs in this important arena. It is also possible that EBP represents simply another conceptual fad which will enjoy a brief flurry of interest, and then fade from view. We have had many examples of this latter scenario. Time will tell.

73

Summary

When the primary sources describing EBP are consulted, it is troubling to see the numerous misconceptions that are being promulgated about this potentially useful model. Autism stakeholders are urged to acquaint themselves with this approach, make their own informed decisions as to its usefulness, and take steps to adopt it, if moved to do so. EBP represents the most sophisticated model to date that has been developed to guide our practice and improve the services we provide.

74

Copies of this powerpoint presentation are available from the author, via Bthyer@fsu.edu

Bruce Thyer, Ph.D., LCSW, BCBA-D College of Social Work Florida State University Tallahassee, FL 32306 USA