Practical Strategies for Improving the Frequency and Quality of - PDF document

Practical Strategies for Improving the Frequency and Quality of Direct Observation of Learners Jennifer R. Kogan, MD Professor of Medicine Assistant Dean, Faculty Development Director Undergraduate Education, Department of Medicine Disclosure I have no relevant commercial interests to disclose. 2 Page 1 1

Workplace Based Assessment  Assessment of day-to-day practice in the authentic clinical environment  Assessment of what doctors actually do in practice Swanwick T. Br J Hosp Med .2009;70:290-3 3 4 Page 2 2

Miller’s Assessment Pyramid Expertise Direct Observation/ WBA Authenticity DOES Standardized patients SHOWS HOW Problem based learning KNOWS HOW Multiple choice exam KNOWS Miller GE. Acad Med. 1990; 65:S63-7 5 Goal To improve the frequency and quality of WBA in your training program 6 Page 3 3

Roadmap  Review the rationale for direct observation  Identify barriers to frequent, high quality WBA  Describe strategies to improve WBA 7 Video Exercise Watch the following encounter between a patient and a PGY-2 resident  List areas performed well  List errors/deficiencies  How would you supervise this resident the next time?  Circle a mini-CEX rating for counseling 8 Page 4 4

Roadmap  Review the rationale for direct observation  Identify barriers to frequent, high quality WBA  Describe strategies to improve WBA 9 Rationale for WBA 1. Clinical skills matter but are variable 2. Central in CMBE 3. Essential for deliberate practice 4. Necessary for effective supervision 10 10 Page 5 5

1: Clinical Skills Matter  History leads to diagnosis > 80% of time  Even in era of technology  Required to avoid unnecessary testing  Faulty data gathering  Source of diagnostic errors  Common cause of death Hampton JR et al. BMJ 1975; 2(5969):486-9 Peterson MC et al. West J Med . 1992; 156(2):163-5 Graber, M et al . Acad Med . 2002;77(10):981-92 National Academy of Medicine. Improving Diagnosis in Medicine , 2015 Makary MA et al. BMJ. 2016; 353:i2139 doi: 10.1136/bmj.i2139 11 11 Patient Centered Care Matters  Improves communication  Promotes patient involvement in care  Increases patient knowledge/self-efficacy  Creates positive relationships with provider  Improves adherence  Improves well-being  Improves outcomes  Decreases costs Levinson W et al. 2010; Health Aff 29: 1310-18 Williams S et al. Fam Prac .1998;15:480-92 DiMatteo M. Patient Educ Counsel . 2004;55:339-44 Stewart M . CMAJ . 1995; 152:1423-33 Vermeir P et al. Int J Clin Pract . 2015;69:1257-1267. 12 12 Page 6 6

Variable Clinical Skills  Graduating students/starting interns  Practicing physicians Stillman. Ann Intern Med.1990; Sachdeva. Arch Surg.1995; Lypson.Acad Med.2004; Wilson. Acad Med 2002; Fox. Med Educ 2002; Mangione. JAMA1997; Braddock.1999 13 13 Direct Observation to Assess Core Skills ASSESSMENT Legitimizes the skills Ensures assessment of essential skills LEARNING 14 14 Page 7 7

2: Essential in CBME Structure Outcome Process (Competence) Time Entrustment 15 15 3: How People Become Experts  Deliberate Practice  Working on well defined tasks  Informative feedback  Repetition  Self-reflection  Motivation  Endurance 16 16 Page 8 8

Coaching Requires Direct Observation 17 17 4: Supervision “. . . closer supervision leads to fewer errors, lower patient mortality, and improved quality of care. . ..” 18 18 Page 9 9

Assessing Does THE PATIENT 19 19 Observation and Safe Patient Care Safe, effective patient-centered care Appropriate level of supervision** **a function of attending competence in context Trainee performance* * a function of level of competence in context Kogan JR et al. Acad Med ; 2014;89:721-7 20 20 Page 10 10

Roadmap  Review rationale for direct observation  Identify barriers to frequent, high quality WBA  Describe strategies to improve WBA 21 21 Lack of Direct Observation: Faculty  Lack of time  Lack of buy-in  Fear of undermining learner-patient relationship  Low self-efficacy  Content/skills assessed  Standards  Feedback approach  Diagnosing learner/offering action plan 22 22 Page 11 11

Lack of Observation: Learners  Lack of faculty time  Lack of trust, longitudinal relationships  Anxiety provoking  Artificial, “check-box” activity  Threatens autonomy and efficiency  Tension feedback vs high stakes assessment Berendonk C et al. Adv Health Sci Educ . 2013;18:559-71 Kogan JR et al. Med Educ. 2011;45:1048-60 Kogan JR et al. Med Educ. 2012;46:201-15 Govaerts MJB et al. Adv Health Sci Educ Theory Pract. 2011;16:151-65 Hatala R et al. Med Teach . 2008;30:414-19 Bing You RG et al. JAMA . 2009;302:1330-1 Watling C et al. Med Educ . 2012;46:593-60 23 23 Low Quality Assessments  Poor accuracy  Variability  Focus of clinical performance  Expectations about what is acceptable  Rating errors  Halo/“Horn” effect  Leniency/stringency effect  Cognitive bias 24 24 Page 12 12

Review Ratings of Video 25 25 Poor Inter-rater Reliability 26 26 Page 13 13

Frames of Reference 1 2 3 4 5 6 7 8 9 Unsatisfactory Satisfactory Superior ORDINAL Below At Expectation Exceeds Expectation Expectation NORMATIVE ???? ???? ???? GESTALT Kogan JR et al. Med Educ. 2011;45:1048-60 27 27 1 2 3 4 5 6 7 8 9 Not Close to What I Do What I Do What I Do (or better) SELF Missing evidence Most evidence All evidence based elements based elements based elements BEST PRACTICE Kogan JR et al. Med Educ. 2011;45:1048-60 28 28 Page 14 14

29 29 Assessors’ Own Clinical Skills Variable, sometimes deficient  History taking  Physical exam  Counseling/shared decision making  Patient centered communication Ramsey PG et al. JAMA 1993;269:1655-60 Paauw DS et al. JAMA 1995;274:1380-2 Vukanovick-Criley JM et al. Arch Intern Med. 2006;166:610-16 Braddock CH 3 rd et al. J Gen Intern Med. 1997;12(6):339-45 Kogan JR. et al. Acad Med. 2010;85(S10):S25-8 Levinson W. BMJ Qual Saf 2011;20:823-5 Frankel RM et al. Acad Med 2011;86:445-52 30 30 Page 15 15

Faculty Skills and Ratings of Learners  Faculty with higher history and patient satisfaction performance scores provide more stringent ratings. Kogan JR. et al. Acad Med. 2010;85(10 Suppl):S25-8 31 31 High Level Inference  Inferences are not recognized  Inferences are rarely validated for accuracy  Inferences can be wrong Govaerts MJB et al. Adv Health Sci Educ Theory Pract. 2011;16:151-65 Kogan JR et al. Med Educ. 2011;45:1048-60 32 32 Page 16 16

Contextual Factors  Encounter complexity  Resident characteristics  Institutional culture Kogan JR et al. Med Educ. 2011;45:1048-60 Kogan JR et al. Med Educ. 2012;46:201-15 33 33 Roadmap  Review rationale for direct observation  Identify barriers to frequent, high quality WBA  Describe strategies to improve WBA 34 34 Page 17 17

Strategies  Getting faculty buy-in  Finding time  Minimizing interference  Improving observation quality  Agreeing on the standard  Activating learners  Maximizing the system 35 35 Strategies  Getting faculty buy-in  Finding time  Minimizing interference  Improving observation quality  Agreeing on the standard  Activating learners  Maximizing the system 36 36 Page 18 18

Who Watched You? Being Observed •How did it feel? •Was it useful? Being the Observer •How did it feel? •Was it useful? 37 37 Highlighting Benefits of Observation 38 38 Page 19 19

Four Arguments for WBA  Clinical skills matter but are variable  Central in CMBE  Essential for deliberate practice  Necessary for effective supervision 39 39 Strategies  Getting buy-in  Finding time  Minimizing interference  Improving observation quality  Agreeing on the standard  Activating learners  Maximizing the system 40 40 Page 20 20

Solution: Observation Snapshots What direct observation snapshots can you identify? *** GOAL: HELP THE LEARNER AND THE PATIENT *** 41 41 Observation Two and Three for Ones  2-for-1 (saves time)?  3-for-1 (saves time + helps patient)? 42 42 Page 21 21

Example Snapshots PHYSICAL PROCEDURES INTERVIEW COUNSELING EXAM  Agenda  Post-rounds  Part of exam  Consent setting  Discharge instructions  Pre-rounds  Procedure  Part of history  Starting medication  Post-check  Pre-rounds  Behavioral change  Family meeting  Code status  Pre-rounds 43 43 CD/PD Helps Prioritize Skills 44 44 Page 22 22

Divide and Conquer 45 45 Strategies  Getting buy-in  Finding time  Minimizing interference  Improving observation quality  Agreeing on the standard  Activating learners  Maximizing the system 46 46 Page 23 23

Minimize Interference  Be a “fly on the wall”  Avoid eye contact with patient  Redirect eye contact Triangulation 47 47 Strategies  Getting buy-in  Finding time  Minimizing interference  Improving observation quality  Agreeing on the standard  Activating learners  Maximizing the system 48 48 Page 24 24