Donald J. Sefcik, DO, MBA Senior Associate Dean College of - - PowerPoint PPT Presentation

▶
donald j sefcik do mba
SMART_READER_LITE
LIVE PREVIEW

Donald J. Sefcik, DO, MBA Senior Associate Dean College of - - PowerPoint PPT Presentation

Dec 30, 2023 244 likes •699 views

Donald J. Sefcik, DO, MBA Senior Associate Dean College of Osteopathic Medicine Michigan State University In-Service Examination (ISE) scores What do they represent? How should you interpret them? What should you do with them?

slide-1
SLIDE 1

Donald J. Sefcik, DO, MBA

Senior Associate Dean College of Osteopathic Medicine Michigan State University

slide-2
SLIDE 2
  • In-Service Examination (ISE) scores

What do they represent? How should you interpret them? What should you do with them?

  • Techniques used to raise ISE scores
  • Recommendations
slide-3
SLIDE 3

Emergency Medicine

  • Hern et al (2009); Cheng (2008); Gillen (1997)

Family Medicine

  • Sevensma SC, Navarre G, Richards RK (2008)

Internal Medicine

  • McDonald, Zeger & Kolars (2008)
  • Garilbaldi et al (2002)

OB/Gyne

  • Withiam-Leitch & Olawaiye (2008)

Pediatrics

  • Aeder, Fogel & Schaeffer (2010)
  • Langenau, Fogel & Schaeffer (2009)

Radiology Surgery

  • Shellito et al (2010); Subhas et al (2009); Kosir et al (2008)
  • Ferguson & Warshaw (2006); Virgilio et al (2003); Bull et al (2001)
slide-4
SLIDE 4

A 72 year-old patient presents with progressive difficulty breathing, jugular venous distention, orthopnea, bilateral lower extremity edema , and bilateral rales. Which of the following is most important when deciding upon treatment options?

  • A. minute respiration
  • B. heart rate
  • C. degree of lower extremity edema
  • D. respiratory rate
  • E. none of the above
slide-5
SLIDE 5

A 72 year-old patient presents with progressive difficulty breathing, jugular venous distention, orthopnea, bilateral lower extremity edema , and bilateral rales. Which of the following is most important when deciding upon treatment options? We don’t treat patients without understanding the etiology of their predicament and the nature of the

  • intervention. Guiding residents’ learning should follow

similar guidelines.

slide-6
SLIDE 6

XR = XT + ME

slide-7
SLIDE 7

XR = XT + ME

XR = Reported ISE score XT = True Score ME = Measurement Errors

slide-8
SLIDE 8

XR = XT + ME

XR = Reported ISE Score Scores

Percent = items correct = raw score Percentile = resident rank compared to peers

slide-9
SLIDE 9

The report of the performance of a PGY2 on his ISE reveals a percent correct of 63 and a percentile rank of 67. What should you do?

  • A. mandate the resident enter a remediation program
  • B. reduce work hours by 25% to increase study time
  • C. do not promote the resident to the PGY3 year
  • D. encourage the resident to keep studying
  • E. resign as program director
slide-10
SLIDE 10
  • What is an “average” percent score?

http://www.eric.vcu.edu/home/curriculum/print/2009Histograms.pdf

slide-11
SLIDE 11
  • What is an “average” percent score?

2009 IM (n = 402 programs) PGY1 = 54% PGY2 = 59% PGY3 = 63%

http://www.eric.vcu.edu/home/curriculum/print/2009Histograms.pdf

slide-12
SLIDE 12
  • What is an “average” percent score?

2009 IM (n = 402 programs) PGY1 = 54% PGY2 = 59%(example was PGY2 = 63%) PGY3 = 63%

http://www.eric.vcu.edu/home/curriculum/print/2009Histograms.pdf

slide-13
SLIDE 13

Your PGY2 from the previous example is now a PGY3. His percentile rank while in your program has been PGY1 = 62, PGY2 = 67, PGY3 = 64. What should you do?

A.

realize that this is a common pattern of progression

B.

worry that this resident will fail his certifying exam

C.

do not promote the resident to the PGY4 year

D.

terminate this resident from your program

E.

delegate the situation to your associate PD

slide-14
SLIDE 14

Why is percentile important?

Increased risk of failing the certifying exam %F 30th 50th 100th

slide-15
SLIDE 15

Why is percentile important?

Increased risk of failing the certifying exam %F

The lower the percentile rank – The greater the risk of failing the certifying examination

30th 50th 100th

slide-16
SLIDE 16

Percent Correct Percentile

Increases each year PGY1 PGY2 PGY3 PGY4

slide-17
SLIDE 17

Percent Correct Percentile

Potential Problem PGY1 PGY2 PGY3 PGY4

slide-18
SLIDE 18

Percent Correct Percentile

Tends to be stable PGY1 PGY2 PGY3 PGY4

slide-19
SLIDE 19

Percent Correct Percentile

Potential Problem PGY1 PGY2 PGY3 PGY4

slide-20
SLIDE 20

Tracking percentiles

  • Three different residents
  • Three years of scores

%F

A1 B1 C1

30th 50th 100th

slide-21
SLIDE 21

Tracking percentiles

  • Three different residents
  • Three years of scores

%F

A1 A2 B1 B2 C1 C2

30th 50th 100th

slide-22
SLIDE 22

Tracking percentiles

  • Three different residents
  • Three years of scores

%F 50th 30th

A1 A3 B3 C1 B1 C3 C2 B2 A2

100th

slide-23
SLIDE 23

Tracking percentiles

  • Three different residents
  • Three years of scores

%F

What can you do to assess the residents between the annual ISE’s?

100th 50th 30th

A1 A3 B1 B3 C1 C3 C2 B2 A2

slide-24
SLIDE 24
  • Make decisions about
  • individual’s educational progression
  • program content areas that are outliers
slide-25
SLIDE 25
  • Make decisions about
  • individual’s educational progression
  • program content areas that are outliers
  • Do not make decisions about:
  • promotion or termination
slide-26
SLIDE 26

XR = XT + ME

XT = True Score = two components

slide-27
SLIDE 27

XR = XT + ME

XT = True Score = two components XC = Score based on content XB = Score based on beyond content

slide-28
SLIDE 28

XR = XC + XB + ME

XR = Reported ISE Score XC = Score based on content XB = Score based on beyond content ME = Measurement Error

slide-29
SLIDE 29
  • Choosing your intervention options
  • Content
  • Beyond Content
slide-30
SLIDE 30

XC

+

XB

Content Beyond Content

PGY1 PGY2 PGY3 PGY4

slide-31
SLIDE 31
  • Assigned / Programmed Reading
  • Conference Attendance
  • Didactic lectures
  • Grand Rounds / M&M
  • Practice Questions
  • Board Review Courses
slide-32
SLIDE 32
  • Assigned / Programmed Reading
  • Conference Attendance
  • Didactic lectures / Journal Clubs
  • Grand Rounds / M&M
  • Practice Questions
  • Board Review Courses
slide-33
SLIDE 33
  • Self-directed Studying
  • Preparing Lectures for Peers
  • Reading Fluency and Accuracy
  • Test Anxiety
slide-34
SLIDE 34
  • Self-directed Studying
  • Preparing Lectures for Peers
  • Questions from / Clarification by an Attending
  • Reading Fluency and Accuracy
  • Developing theme in repeat failures
  • Test Anxiety
  • Definite impact on memory / recall
slide-35
SLIDE 35
  • Intervention options
  • Content = more impact earlier in training

PGY<2 - Programmed Reading / Didactics PGY>3 - Programmed Reading / Didactics

slide-36
SLIDE 36
  • Beyond Content
  • Resident Heterogeneity (all years of training)

Studying Skills Test-taking Skills

slide-37
SLIDE 37
  • Beyond Content
  • Resident Heterogeneity

Studying Skills

  • Memory techniques
  • Reading and Note-taking
  • Transfer

Test-taking Skills

  • Reading- Fluency and Accuracy
  • Test Anxiety
  • Test-Wiseness
slide-38
SLIDE 38

1 / 6

slide-39
SLIDE 39

2 / 6

slide-40
SLIDE 40

3 / 6

slide-41
SLIDE 41

4 / 6

  • 1. Assess the resident’s situation

Develop an individualized plan It’s not just about XC

  • 2. Create a team

Resident and Attending Attending = Coach to Mentor

  • 3. Establish a schedule

Regular meetings

  • 4. Monitor and Refine
slide-42
SLIDE 42

5 / 6

  • 1. Assess the resident’s situation

Develop an individualized plan

There are

It’s not just about XC

resources

  • 2. Create a team

to help you

Resident and Attending

help your

Attending = Coach to Mentor

resident

  • 3. Establish a schedule

Regular meetings

  • 4. Monitor and Refine
slide-43
SLIDE 43

6 / 6

We need to transition away from our focus on teaching (faculty-centered) and instead begin to focus on learning (resident-centered).

slide-44
SLIDE 44

Are there some Questions?

Complete Reference List will be posted (pdf)

  • n SCS website

Emergency Medicine

  • Hern et al (2009); Cheng (2008); Gillen (1997)

Family Medicine

  • Sevensma SC, Navarre G, Richards RK (2008)

Internal Medicine

  • McDonald, Zeger & Kolars (2008)
  • Garilbaldi et al (2002)

OB/Gyne

  • Withiam-Leitch & Olawaiye (2008)

Pediatrics

  • Aeder, Fogel & Schaeffer (2010)
  • Langenau, Fogel & Schaeffer (2009)

Radiology Surgery

  • Shellito et al (2010); Subhas et al (2009); Kosir et al (2008)
  • Ferguson & Warshaw (2006); Virgilio et al (2003); Bull et al (2001)