Mapping to the Milestones A CONCEPTUAL PRESENTATION ON HOW TO THINK - - PDF document

3/14/2016 1 Mapping to the Milestones

A CONCEPTUAL PRESENTATION ON HOW TO THINK ABOUT LINKING YOUR EVALUATION TOOLS, HAVING YOUR EVALUATION TOOLS WORK FOR YOU AND NOT AGAINST YOU

Jennifer W. Swoyer D.O. Osteopathic Program Director/Director

• f Medical Education

Adventist La Grange Family Medicine Residency Program What are the Milestones

 The Milestones are an ACGME mandated mapping document  They must be completed twice yearly on each resident in the

program

 The Milestone document must be completed by a reviewing

committee called the CCC committee

 The document itself is NOT an evaluation  The data collected from each program will be reviewed by

the ACGME over the course of 10 years…

3/14/2016 2

UGH….what does that all mean

 As more and more pressure arises to ensure that physician education is

standardized/nationalized the Milestone project becomes an

• pportunity for programs to evaluate each resident against a

standardized norm of expected behaviors. These behaviors are the substrates of the general competencies that we have worked to integrate over the last 10+ years.

 The milestone document gives a standard map of expected physician

behaviors that we can map against. This ultimately will highlight the success of our residents and our training programs

“ ”

Until we changed our evaluation system, the end of the year evaluations were always due in May. I hated May. I live in Chicago, May is the only reason I stay.

UNNAMED OVERBURDENED PROGRAM DIRECTOR

How Many of you dislike your current evaluation system?

“ ”

How many of you want an evaluation system that will work for you?

Then Stay seated…

3/14/2016 3 So, lets step back and look at the big picture…

Let’s remember…Our Ultimate Goal

Competent physician

So How do we get there? How did we get here?

3/14/2016 4

Competent physician Medical school graduate Competent in 13 standard EPA’s, ready for residency Residency program curriculum Residency Teaching modalities and rotations Residency Program Evaluation System Milestone Sub-competencies Competencies Physician Leadership Professional Development Board Re-Certification Maintenance of Certification Continuing Medical Education Licensure JWS-LGFPR 8-15-15

A Model of Physician Development

As Residency Programs we are accountable for documenting and assessing a young physician’s competence

Competent physician Medical school graduate Competent in 13 standard EPA’s, ready for residency Residency program curriculum Residency Teaching modalities and rotations Residency Program Evaluation System Milestone Sub-competencies Competencies Physician Leadership Professional Development Board Re-Certification Maintenance of Certification Continuing Medical Education Licensure JWS-LGFPR 8-15-15

A Model of Physician Development

3/14/2016 5

We used to think this meant….

Resident evaluates faculty

• n all competencies

Everyone evaluates everyone faculty evaluates every component everytime Resident evaluates all rotations Every evaluator evaluates on all competencies Peer evaluations

Unfortunately our old system had a lot

• f input and little output

3/14/2016 6

We started by correcting our vision

We Put Competence at the center. Competence is our target.

COMPETENCE

Competent Physician

OSTEOPATHIC PRACTICE AND PRINCIPLES OMM/OPP COMMUNICATION C PATIENT CARE PC SYSTEMS BASED PRACTICE SBP PRACTICE BASED LEARNING PBLI MEDICAL KNOWLEDGE MK PROFESSIONALISM P

We added the competencies

Competent physician

OMM OPP C PC SBP PBLI P MK MK2 MK1 PC1 PC2 PC3 PC4 PC5

OUR VISION GREW TO INCLUDE THE SUB COMPETENCIES

3/14/2016 7

Competent physician

OMM & OPP MK PBLI IC SBP PC PROF

ITE

Next we layered

• n the rotations

and educational experiences where we could collect data on resident performance

And the work had just begun…

 Evaluating where we could collect our data was a group process with

all of the faculty

 Our objective was to get multiple data points for each sub-

competency

 When we finished… we had a list of the evaluations we needed to

create.

Evaluations were linked to sub- competencies

C-1: FMC, PEDS, OB, Geri, OSCE, Patient evaluation C-2: FMC, OSCE, shadowing, video, FMS C-3: FMS, PEDS, nursing eval, Patient evaluation C-4: FMS, FMC, nursing eval, Patient evaluation P-1: all rotations, front desk eval, nursing eval, FMS P-2: Coordinator eval, front desk eval, OSCE P-3: FMC, Behavioral health, FMS, OSCE, Patient evaluation P-4: Advisor and PD Review OMM: FMS, FMC, OMT clinic eval

3/14/2016 8

 Front desk Evaluation of Resident: P-1, P-2a, P-2b, P-2c  Journal Club: PBLI-1  Nursing Evaluation of Resident: C-2, C-1, C-3, P-2, P-3, P-2, PC-4, PBLI-3  Outpatient Clinic Evaluation: C-1, C-2, C-3, MK-2, P-1, P-2, OMM, P-2, P-3,

PC-1a, PC-1b, PC-2a, PC-2b, PC-3a, PC-3c, Pc-4, Pc-5, SBP-1, SBP-2, SBP-4, PBLI-1

Before we were done, our bullseye grew

Competent physician

OMM & OPP MK PBLI IC SBP PC PROF

Residency Curriculum and Curricular Objectives Residency Teaching Modalities and Rotations Residency Evaluation System

ITE

3/14/2016 9

An evaluation plan meant

NEW EVALUATIONS

We created a system of standardized performance statements

 These linked to the sub-competencies  Some sub-competencies required several

statements to absorb the important elements

“ ”

Milestone: Patient Care (PC-3) “Partners with the patient, family and community to improve health through disease prevention and health promotion”

3/14/2016 10

We turned this into…

PC-3a

Partners with the patient and family to

• vercome barriers to

disease prevention and health promotion. Is able to describe cost benefit ratio in relation to health promotion and disease prevention therapies

PC-3b

Tracks and monitors disease prevention and health promotion within the practice.

PC-3c

Reconciles the various guidelines and screening recommendations for health maintenance and applies these appropriately to patient care.

We completed this process for all of the competencies

 This gave us a list of evaluation statements that linked to the sub-

competencies.

 With this list we could build our new evaluations tools and link them

directly to the sub-competencies and build a model for completing our Milestone documents.

 There was just one piece missing… One major problem

Please raise your hand if you are below average.

Physicians by nature are over achievers. We tend to be above

• average. We are not always great evaluators.

3/14/2016 11

Has anyone here suffered the tears and anguish of an average resident who was given an average evaluation? Let’s say a 3/7 Likert scale?

There had to be a better scale… The Milestone system is a 9 point scale. We chose 5 major performance definitions



1 = Beginner: This resident is performing at a level that shows awareness of the medical system but he/she depends on other physicians for the development of patient care plans

 2 = Advanced Beginner: This resident is performing at a level that

demonstrates awareness of the medical system and patient care. He/she is developing assessments of the patients in his/her care and utilizing attending input for the development of care plans.

 3 = Intermediate: This resident is performing at a level that demonstrates

independent thought. This resident is able to evaluate, assess and develop a plan of care for most patients.

3/14/2016 12

 4 = Independent: This resident is performing at a level that demonstrates

the ability to teach others and function in independent practice. This resident is able to evaluate, assess and develop a plan of care for all patients.

 5 = Expert: This resident is performing at the highest level, consistent with

experts in the field of family medicine. This resident is able to evaluate the medical system and current models of care. This resident contributes to the development of systems that impact patient care within the

• community. He/she role models active involvement in community

education and health policy.

“ ”

This scale has significantly reduced grade inflation

Especially from the specialists 

Every subcompetency statement was entered in New Innovations

3/14/2016 13

Evaluations were built using New

• Innovations. We linked each

question to the corresponding competency

3/14/2016 14 A new evaluation tool was generated….

Once all of our evaluations were created, we developed a spread sheet to ensure all sub-competencies were measured several times during the course of the year.

ARE YOU EXCITED YET???

3/14/2016 15

Once we were sure… we covered all sub-competencies we had all evaluations written we had all evaluations linked to the Milestones we developed a schedule for evaluation completion Then we had our first CCC meeting.

3/14/2016 16

OMG!!!! It works!!! For Our CCC meeting for the December Milestones

 The committee met using the following information  Resident self evaluations (where they filled out a Milestone document on themselves)  The New-Innovations generated averaged Sub-competencies  Faculty discussion and impressions  The negotiated Milestone was completed and entered into New-Innovations

3/14/2016 17

Each resident’s data will correlate to a graph of their milestone performance

3/14/2016 18 The completed Milestone data set gives us a graphic representation of Milestone performance.

And, ultimately…measured and documented levels of competence.

Thank you to my amazing coordinator Becky! Without her help this presentation would never have been completed, or have graphics and color.

And Good Luck to you.

3/14/2016 19

If you would like to discuss our tools or evaluations, or copies, please contact us. Jennifer.Swoyer@ahss.org

• r

Rebecca.Webb@ahss.org