Us Using ng evaluat luation ion data a to improve the desi sign - - PowerPoint PPT Presentation

Us Using ng evaluat luation ion data a to improve the desi sign gn and d implementation of an annual CV fellows’ training progra

• gram

2017 7 As Assoc sociati iation

• n of Progra

ram m Direct ctor

• rs in Vasc

scular ular Sur urger gery y Spring ing Meet eting ing March h 31, 1, 2017 Rosemo semont, nt, IL

Ebun n Ebunlomo, nlomo, PhD, MPH, MCHES ES Melanie Lazarus, MPH Mariana Smith, BA Jean Bismuth, MD

• C. Huie Lin, MD, PhD, FACC

Ross Reul, MD Alan B. Lumsden, MD

Disclosures

• None

Ba Back ckgr ground

• und
• Residents have variable knowledge of cardiovascular

diseases and technical skills

• Gaps in knowledge and skills have significant

implications for clinical outcomes

• It is important to evaluate the usefulness of current

training courses for first- and second-year fellows

DeBakey Institute for Cardiovascular Education & Training

CV Fellows’ Boot Camp

• 3 tracks: cardiology, cardiac

surgery and vascular surgery

• 90 learners
• 55 faculty
• 33 hands-on stations
• 90 presentations
• 3 social events

• Enhances the utili

lity ty and actual us use of evaluations

• Focuses on intended users
• f resultant data
• Aligns data collection tools

with pre-determined evaluation questions

Uti tilit lity-Focused

• cused Ev

Evaluation luation

Uti tility lity-Focused

• cused Ev

Evaluat luation ion

PURPOSE E OF EVALUATION ION

• Identify strengths and weaknesses to make improvements for future program offerings
• Determine progress toward program learning outcomes

USERS RS AND USES EVALUATIO TION QUESTIO TIONS INDICATORS RS DeBakey y Educa cati tion n Team/DeBa Bakey y Heart and Vascular r Center r Leadership/ ship/ Course Direct ctor

• rs:

s:

• Determine effectiveness of CV

Fellows’ Boot Camp to guide improvements

• Make decisions regarding

additional educational needs or interests

• Determine whether or not CV

Fellows’ Boot Camp is a good use

• f resources

To what degree did participants react favorably to the learning event? (Level 1: Reaction) To what degree did participants acquire knowledge, skills and attitudes based on their participation in the learning event? (Level 2: Learning) Average score on post-training survey items Number of positive comments about the program Number of recommendations for areas of improvement for future programming Proportion of participants that could list at least 3 useful pieces of information acquired from the training

• Session-based survey
• On-site observations

Da Data ta Collection lection Tools ls

• Post-training summative survey
• Post-event debrief meeting

CONT NTENT ENT Strongl

• ngly

Agree ee Agree ree Not

• t sure Disag

agre ree Strongl

• ngly

Disagre agree 1. This session was informative and relevant.      1. The duration was adequate to cover all session topics.      1. This session increased my understanding of the subject matter.      SPEAK EAKERS ERS Strongl

• ngly

Agree ee Agree ee Not

• t sure Disagre

agree Strongl

• ngly

Disagre agree Anatomy & Physiology [PRESENTER’S NAME] was an effective presenter.      Diagnosis & Imaging of Visceral Artery Disease [PRESENTER’S NAME] was an effective presenter.      Open & Endovascular Management of Occlusive Disease [PRESENTER’S NAME] was an effective presenter.      Management of Visceral Artery Aneurysms [PRESENTER’S NAME] was an effective presenter.      Overall, I am satisfied with all aspects of this session.     

Track ack Respo spons nse e Rate Cardiac Surgery 44% Cardiology 53% Vascular 63%

• Hands-on portion was more

beneficial compared to the didactic courses

• Need for more hands-on

training to translate their newly acquired knowledge and skills into practice

Da Data ta High ghlight lights

Fell llowsh ship p Year ar Frequenc uency 1st 39 2nd 9 4th year integrated Resident 1 Not recorded 3 Total 52

Par Participant ticipant Sur urvey y Da Data ta

CATEG EGOR ORY AVERAGE E RATING COMMENT ENTS

Content 5 (Excellent)

• Cadaver dissection was good--expand on this.
• Lots of content, may be helpful to get slide packet before
• Good review of common topics as well as less commonly

encountered pathologies

• Very much educational

Format 5 (Excellent)

• Wish some lectures had more time
• Format gave nice overview
• Excellent!
• Didactic lectures not always the best format.

Speakers 5 (Excellent)

• Eidt, Azzizadeh, Bechara, Baltazar, Lumsden were excellent.
• Drs. Eidt and Lumsden were excellent.
• I liked the ones that referred to studies for their decision making.
• All great!
• Excellent speakers, all very knowledgeable on their topics
• Many speakers had too many slides for the time allotted

Usefulness 5 (Excellent)

• Very good exposure to all spectrum of vascular surgery
• Great!
• Very good starting point for where I need to focus my reading

Facility/Location 5 (Excellent)

• Simulation lab/cadaver lab were excellent
• Exceptional location and arrangements

Support Staff 5 (Excellent)

• Ms. Smith has been extremely helpful
• Event seemed very well organized
• Amazing, very helpful, friendly staff

Ski kills lls Academ cademy y Da Data ta

• 90% of trainees were comfortable with getting radial access post-training
• Significant improvement in transradial access (TRA) skills (p < 0.05)

• Increase the amount of time for hands-on training

Da Data ta-driv driven en De Decis cisions ions

13 13 8

2016 6 Bootcam tcamp p Traini ning ng Hours

Dida Didactic Co ctic College lege Hands ands-o

• on

n Sk Skills ills Acade Academy my 9 16 16 2017 7 Bootca tcamp p Train ining ing Hour urs Dida Didactic Co ctic College lege Ha Hands-o nds-on n Sk Skills Acade ills Academy my

Formalize session moderation

– Establish a DeBakey Education staff moderator for each session – Provide faculty moderator with pre-Boot Camp training and a guiding document to foster effective facilitation of learning

Da Data ta-driv driven en De Decis cisions ions

Align contents with specific learning outcomes

– Add learning objectives to guide preparation of training contents, faculty selection, synthesis of knowledge by facilitator, and evaluation

Refine process flow

– Collate all participant materials and distribute upon registration – Allow ample time for breaks and evaluation

Da Data ta-driv driven en De Decis cisions ions

• Start with the end in mind

to ensure data utility

• Engage intended users

during evaluation planning

• Align data collection tools

with intended users/uses

• Use evaluation data to

reflect, improve and revise

Sum umma mary

Ref eferences erences

1. Bismuth, J, Duran, C, Donovan, M, Davies, MG & Lumsden, AB (2012). The Cardiovascular Fellows Bootcamp.' Journal of vascular surgery, Vol 56, no. 4. 2. Kirkpatrick D.L. (1959). Techniques for evaluating training

• programs. Journal of American Society of Training Directors. Vol 13, no. 3:
• pp. 21–26.

3. Mitchell, E.L., Arora, S., Moneta, G.L. (2011). Ensuring vascular surgical training is on the right track. Journal of vascular surgery, Vol 53, pp. 517– 525 4. Patton, M.Q. (2008). Utilization-focused evaluation, 4th edition. Thousand Oaks, CA: Sage. 5. Patton, M.Q. and Horton, D. (2009). Utilization-Focused Evaluation for Agricultural Innovation. Institute of Learning and Change (ILAC) Brief No.

• 22. ILAC, Bioversity, Rome.

Contact DeBakey Education

Ebun O. Ebunlomo, PhD, MPH, MCHES Curriculum Developer Houston Methodist DeBakey Heart & Vascular Center 713-363-9336 eebunlomo@houstonmethodist.org www.houstonmethodist.org/dicet

DeBakey Institute for Cardiovascular Education & Training