Us Using ng evaluat luation ion data a to improve the desi sign - PowerPoint PPT Presentation

Us Using ng evaluat luation ion data a to improve the desi sign gn and d implementation of an annual CV fellows’ training progra ogram 2017 7 As Assoc sociati iation on of Progra ram m Direct ctor ors in Vasc scular ular Sur urger gery y Spring ing Meet eting ing March h 31, 1, 2017 Rosemo semont, nt, IL Ebun n Ebunlomo, nlomo, PhD, MPH, MCHES ES Melanie Lazarus, MPH Mariana Smith, BA Jean Bismuth, MD C. Huie Lin, MD, PhD, FACC Ross Reul, MD Alan B. Lumsden, MD

Disclosures • None

Ba Back ckgr ground ound • Residents have variable knowledge of cardiovascular diseases and technical skills • Gaps in knowledge and skills have significant implications for clinical outcomes • It is important to evaluate the usefulness of current training courses for first- and second-year fellows

CV Fellows’ Boot Camp DeBakey Institute for Cardiovascular Education & Training • 3 tracks: cardiology, cardiac surgery and vascular surgery • 90 learners • 55 faculty • 33 hands-on stations • 90 presentations • 3 social events

Uti tilit lity-Focused ocused Ev Evaluation luation • Enhances the utili lity ty and actual us use of evaluations • Focuses on intended users of resultant data • Aligns data collection tools with pre-determined evaluation questions

Uti tility lity-Focused ocused Ev Evaluat luation ion PURPOSE E OF EVALUATION ION • Identify strengths and weaknesses to make improvements for future program offerings • Determine progress toward program learning outcomes USERS RS AND USES EVALUATIO TION QUESTIO TIONS INDICATORS RS DeBakey y Educa cati tion n Team/DeBa Bakey y To what degree did Average score on post-training Heart and Vascular r Center r participants react favorably survey items Leadership/ ship/ Course Direct ctor ors: s: to the learning event? • Determine effectiveness of CV (Level 1: Reaction) Number of positive comments Fellows’ Boot Camp to guide about the program improvements To what degree did Number of recommendations participants acquire • Make decisions regarding for areas of improvement for knowledge, skills and additional educational needs or future programming attitudes based on their interests participation in the learning Proportion of participants that event? • Determine whether or not CV could list at least 3 useful (Level 2: Learning) Fellows’ Boot Camp is a good use pieces of information of resources acquired from the training

Da Data ta Collection lection Tools ls Strongl ongly ree Strongl ongly Agree ree Not ot sure Disag agre CONT NTENT ENT Agree ee Disagre agree 1. This session was informative and relevant.      1. The duration was adequate to cover all session topics.      1. This session increased my understanding of the subject      matter. Strongl ongly agree Strongl ongly Agree ee Not ot sure Disagre SPEAK EAKERS ERS Agree ee Disagre agree Anatomy & Physiology      [PRESENTER’S NAME] was an effective presenter. Diagnosis & Imaging of Visceral Artery Disease      [PRESENTER’S NAME] was an effective presenter. Open & Endovascular Management of Occlusive Disease      [PRESENTER’S NAME] was an effective presenter. Management of Visceral Artery Aneurysms      [PRESENTER’S NAME] was an effective presenter. Overall, I am satisfied with all aspects of this session.      • Session-based survey Post-training summative survey • • On-site observations • Post-event debrief meeting

Da Data ta High ghlight lights Fell llowsh ship p Year ar Frequenc uency Hands-on portion was more • 1st 39 beneficial compared to the 2nd 9 didactic courses 4th year integrated Resident 1 Not recorded 3 • Need for more hands-on training to translate their Total 52 newly acquired knowledge and skills into practice Respo spons nse e Track ack Rate Cardiac Surgery 44% Cardiology 53% Vascular 63%

Par Participant ticipant Sur urvey y Da Data ta CATEG EGOR ORY AVERAGE E RATING COMMENT ENTS • Cadaver dissection was good--expand on this. • Lots of content, may be helpful to get slide packet before • Content 5 (Excellent) Good review of common topics as well as less commonly encountered pathologies • Very much educational • Wish some lectures had more time • Format gave nice overview Format 5 (Excellent) • Excellent! • Didactic lectures not always the best format. • Eidt, Azzizadeh, Bechara, Baltazar, Lumsden were excellent. • Drs. Eidt and Lumsden were excellent. • I liked the ones that referred to studies for their decision making. Speakers 5 (Excellent) • All great! • Excellent speakers, all very knowledgeable on their topics • Many speakers had too many slides for the time allotted • Very good exposure to all spectrum of vascular surgery • Usefulness 5 (Excellent) Great! • Very good starting point for where I need to focus my reading • Simulation lab/cadaver lab were excellent Facility/Location 5 (Excellent) • Exceptional location and arrangements • Ms. Smith has been extremely helpful • Event seemed very well organized Support Staff 5 (Excellent) • Amazing, very helpful, friendly staff

Ski kills lls Academ cademy y Da Data ta • 90% of trainees were comfortable with getting radial access post-training • Significant improvement in transradial access (TRA) skills (p < 0.05)

Data Da ta-driv driven en De Decis cisions ions • Increase the amount of time for hands-on training 2016 6 Bootcam tcamp p Traini ning ng Hours 2017 7 Bootca tcamp p Train ining ing Hour urs 9 8 13 13 16 16 Dida Didactic Co ctic College lege Didactic Co Dida ctic College lege Ha Hands-o nds-on n Sk Skills Acade ills Academy my Hands ands-o -on n Sk Skills ills Acade Academy my

Da Data ta-driv driven en De Decis cisions ions Formalize session moderation – Establish a DeBakey Education staff moderator for each session – Provide faculty moderator with pre-Boot Camp training and a guiding document to foster effective facilitation of learning

Da Data ta-driv driven en De Decis cisions ions Align contents with specific learning outcomes – Add learning objectives to guide preparation of training contents, faculty selection, synthesis of knowledge by facilitator, and evaluation Refine process flow – Collate all participant materials and distribute upon registration – Allow ample time for breaks and evaluation

Sum umma mary • Start with the end in mind to ensure data utility • Engage intended users during evaluation planning • Align data collection tools with intended users/uses • Use evaluation data to reflect, improve and revise

Ref eferences erences 1. Bismuth, J, Duran, C, Donovan, M, Davies, MG & Lumsden, AB (2012). The Cardiovascular Fellows Bootcamp.' Journal of vascular surgery , Vol 56, no. 4. 2. Kirkpatrick D.L. (1959). Techniques for evaluating training programs. Journal of American Society of Training Directors . Vol 13, no. 3: pp. 21 – 26. 3. Mitchell, E.L., Arora, S., Moneta, G.L. (2011). Ensuring vascular surgical training is on the right track. Journal of vascular surgery , Vol 53, pp. 517 – 525 4. Patton, M.Q. (2008). Utilization-focused evaluation, 4th edition. Thousand Oaks, CA: Sage. 5. Patton, M.Q. and Horton, D. (2009). Utilization-Focused Evaluation for Agricultural Innovation. Institute of Learning and Change (ILAC) Brief No. 22. ILAC, Bioversity, Rome.

Contact DeBakey Education DeBakey Institute for Cardiovascular Education & Training Ebun O. Ebunlomo, PhD, MPH, MCHES Curriculum Developer Houston Methodist DeBakey Heart & Vascular Center 713-363-9336 eebunlomo@houstonmethodist.org www.houstonmethodist.org/dicet