Innovations in Medical Education Session Moderators: Chayan - - PDF document

Innovations in Medical Education Session Moderators: Chayan Chakraborti, MD and Rita S. Lee, MD Co-Chairs, Innovations in Medical Education, 37th Annual Meeting

SEALS: A PROCESS AND INTERMEDIATE OUTCOME EVALUATION OF AN INNOVATIVE PIPELINE CURRICULUM Cassandra D. Fritz; Monica Vela. University of Chicago, Chicago, IL. (Tracking ID #1936651) NEEDS AND OBJECTIVES: Pipeline programs are one proposed method to increase the number of under-represented minority (URM) students matriculating to medical school. Although pipeline programs have demonstrated success in increasing matriculation

• f URM students, a review conducted by the Department of Health and Human Services in 2003 notes the lack of appropriate

evaluation measures for pipeline curriculum. Therefore, our goals were to 1) develop an evaluation mechanism using process and intermediate outcome measures for pipeline programs and 2) apply the evaluation to a novel pilot pipeline program. SETTING AND PARTICIPANTS: The SEALS summer pipeline program is a novel curriculum established to promote 5 competencies among minority medical students including: Socialization and Professionalism, Education in science learning tools, Acquiring finance literacy, Leveraging Mentorship and Networks, and Social expectations and resilience. Local academically successful URM and/or underserved rising first and second year college students were recruited to apply through the web. IRB approved pre/post program surveys included short answer, likert scale responses, and personal statement writing and were administered to the 13 SEALS students. In addition, this survey was administered to a cohort of ten students in a comparable, research-based pipeline program at our institution during the same summer session. This cohort served as a control. DESCRIPTION: The pre and post survey instrument included qualitative and quantitative measures designed to assess students' knowledge, skills, attitudes, and quantify hours of programmatic activities. Students' knowledge of (1) the medical school admissions process, (2) requirements for medical school, (3) AMCAS application service, (4) Health Care Disparities, and (5) Anatomy and Physiology of systems related to greatest health threats to underserved populations was assessed. Skills of how to (1) shadow a physician effectively, (2) seek out research opportunities, (3) engage and reflect upon service opportunities, (4) develop a personal statement of interest in medicine, (5) interview and communicate, and (6) effectively complete of a financial aid form were also

• assessed. The survey gauged the students' attitudes toward the following: (1) service, (2) research, (3) medicine as a profession, and

(4) commitment to medicine as a career. Finally, the total number of hours spent on each activity and student's assessment of the quality of the activities was obtained. EVALUATION: Our evaluation consisted of process and intermediate outcome measures. Process measures included who participated, quantity of activities, and quality of those activities. Intermediate outcome measures quantified students' change in knowledge, skills, and attitudes. The evaluation was also administered to the control cohort, who did not differ significantly in any demographic when compared to the SEALS cohort. Process Outcomes. 100% of students agreed/strongly agreed that the Meet the Professor series, Health Care Disparities series, and Pre-Med 101 workshops were instrumental in motivating students to purse their passion for medicine (11 of 11). Furthermore, these curriculum components inspired students to advocate for minority health issues. Intermediate Outcomes. The pre-SEALS to pre-control analysis did not reveal any important statistically significant difference between the two cohorts. A comparison of pre-SEALS to post-SEALS found statistically significant changes favoring post- SEALS: knowledge of MCAT components (p=0.007), ability to ask for letter of recommendation (p = 0.04), attitudes toward research being an important determinant of success (p= 0.02) and appearing confident during an interview (p=0.04). The post-SEALS to post-control did uncover a number of statistically significant values including, but not limited to ability to communicate with patients (p=0.046), understanding of health disparities (p=0.04), understanding of major disease entities (p< 0.01), and students' resolve to become a physician (p<0.01). Pre and post- SEALS personal statements were compared to the control group's AMCAS personal statement using a 5-point scale. The SEALS cohort average score increased by nearly 1 point. (pre = 2.09 to post= 2.95). These averages were then compared to the control cohort average, which was determined to be 2.40. DISCUSSION / REFLECTION / LESSONS LEARNED: Two major lessons about pipeline curriculum emerge. First, defining best practices in pipeline curriculum requires process and intermediate outcome measures. Outlining process and intermediate outcome measures of pipeline programs will allow for systematic improvement of program practices and dissemination of successful pipeline

• practices. Second, use of a control group revealed a gap in professional development training for URM students participating in

research-based pipeline programs.

CLOSING THE FEEDBACK LOOP -EVALUATING AN INNOVATIVE SYSTEM TO PROVIDE FEEDBACK TO FACULTY ABOUT THEIR STUDENT ASSESSMENTS Neil Mehta1; Alan L. Hull2; Amy S. Nowacki3. 1Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland, OH; 2Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland, OH; 3Cleveland Clinic, Cleveland, OH. (Tracking ID #1933607) NEEDS AND OBJECTIVES: Students need ongoing formative feedback that identifies specific strengths and areas for improvement, to compare this feedback to established goals and expectations, and to develop plans to improve targeted areas to become self-regulated learners. We have developed a web-based Clinical Assessment System (CAS) that allows all preceptors to provide timely formative feedback based on observed clinical encounters. While finding the time to provide formative feedback is a challenge, a bigger challenge is that clinical preceptors are more familiar with providing summative end-of-rotation feedback and may be unaware of the type of feedback that is most useful to help students identify learning needs, seek new knowledge or skills, and improve performance and understanding. To address this issue, we enhanced CAS to allow students to assess the usefulness of the formative feedback they receive from the clinical preceptors. The objectives of this project are to determine if these assessments can:

• 1. Reliably and validly identify preceptors for faculty development or for recognition; 2. Be aggregated to identify disciplines or

clinical training sites for faculty development. SETTING AND PARTICIPANTS: The study included 48 students from 1 of 3 tracks of a medical school during their core clinical rotations based at 3 medical centers. The track uses a competency-based portfolio system for assessment. This system requires students to write a reflective essay on 9 different competencies citing their formative assessments as evidence of meeting standards in each competency. DESCRIPTION: The students log all their educationally significant patient encounters in CAS using any internet enabled device. The preceptors provide competency-based narrative formative feedback (both strengths and areas for improvement) for each of these

• encounters. When viewing this formative feedback online, students can rate the usefulness of the feedback using a 5-point Likert item
• f agree-disagree to the statement "This form identifies specific behaviors /skills that I did well and/or I can improve upon". The

student rating is presently not shared with preceptors. EVALUATION: Students rated the usefulness of 1,181 (59%) forms out of 2,001 formative assessments completed by faculty between July-December, 2013 as follows: 56% Strongly Agree (SA); 33% Agree (A); 7% Neutral (N); 2% Disagree (D); and 1% Strongly Disagree (SD). 42 forms were rated D or SD. 24 of these 42 forms had non-specific comments (e.g. "good history") rather than specific feedback about observed interactions while 11 had no competency specific comments. Only 4 of these forms were blank. 24 of these forms were from 3 students. Other observations of these 42 forms included faculty copying and pasting the identified criteria into the feedback boxes and faculty who entered a number (e.g. 9/10) instead of comments in the feedback box. Each form rating was dichotomized as Useful (SA/A) versus Not Useful (N/D/SD). When restricted only to disciplines with at least 50 assessed forms, the usefulness of the forms differed according to discipline (73% surgery, 83% aging, 88% pediatrics, 89% OB/GYN, 91% internal medicine, 91% family medicine; Chi-square test p = 0.001). The usefulness of the forms also differed according to clinical site (82% vs. 87% vs. 90%; Chi-square test p = 0.05). Faculty level statistics were computed for the disciplines with the largest number (internal medicine, n = 134) and the most diversely rated (surgery, n = 32) faculty. The majority of these faculty (81%) had 3 or fewer forms rated by students. Points were assigned to each student's assessment of the usefulness of each form they rated: SD = 0, D = 2, N = 5, A = 8, SA = 10; the average for each faculty being the feedback score. The median feedback score was 9.3, Q1 = 8 and Q3 = 10. Nineteen faculty (11%) had average scores below 6 and were identified as providing student-reported non-useful feedback. DISCUSSION / REFLECTION / LESSONS LEARNED: Formative assessments are critical to help students develop into self- regulated learners by letting them close the gap between goals and performance. Providing useful feedback to students is a faculty skill that can be improved. We describe a system for students to assess the usefulness of feedback they receive. Our analysis suggests that these data can be used to identify faculty who could benefit from faculty development on providing feedback and recognize faculty who consistently provide useful feedback. The next step is to validate the results with a different cohort of students doing their clinical rotations at the same sites, in the same disciplines with the same group of faculty. We also plan to compare student and expert assessments of usefulness from a randomly selected group of assessments and study the utility of using student assessments of usefulness as an outcome marker for faculty development. If borne out by these studies, this model can be replicated on other assessment systems and help both students and faculty in academic programs.

STORYTELLING WITH INPATIENTS Katherine C. Chretien1,2; Rebecca Swenson3; Bona Yoon1; Ricklie Julian4; Jonathan Keenan2; Raya Kheirbek1,2. 1Washington DC VAMC, Washington, DC; 2George Washington University, Washington, DC; 3Dartmouth-Hitchcock Medical Center, Lebanon, NH;

4Montefiore Medical Center, Bronx, NY. (Tracking ID #1938337)

NEEDS AND OBJECTIVES: Empathy is an essential component of patient-centered care and improves patient outcomes. A recent systematic review of interventions to teach empathy to medical trainees identified a need for future interventions to be built upon a foundation of relationship-centered care. Narrative competence can be defined as "the ability to acknowledge, absorb, interpret, and act on the stories and plights of others." There are few published educational interventions that address empathy while involving actual patient-provider interactions, and/or address narrative competence. We sought to incorporate a brief experiential narrative medicine curriculum within the third-year medicine clerkship with the goals of developing narrative competence, practicing attentive listening, and stimulating reflection while providing patient-centered care for hospitalized patients. SETTING AND PARTICIPANTS: Single institution with third-year basic medicine clerkship students on a 4 week rotation and inpatients on the acute (non-ICU) medicine wards. DESCRIPTION: The narrative medicine curriculum consisted of 1) Introductory session where students were introduced to narrative medicine and practiced attentive listening to storytelling in pairs; 2) Patient activity where students elicited illness narratives from patients, attentively listened, wrote their version of the story, and then read these back to patients. Students also asked patients to select a piece of art to help tell their story, chosen from a digital collection stored on an iPad. 3) Debriefing/reflection session where students wrote reflectively and participated in a facilitated discussion about their experiences with the activity. EVALUATION: To evaluate and help assess acceptability and feasibility of the curriculum, five focus groups were held between July 2011 and March 2012. Thirty-one students participated. Transcripts were qualitatively analyzed for themes by two independent researchers; disagreements were resolved through discussion. Students discussed their patients' reactions to the narrative medicine activity, students' own experiences, the student-patient dynamic, and challenges. They also identified what they felt they learned, including seeing the patient as human, being open, and clinically relevant insights. Students felt that the activity improved the student- patient relationship and, in some cases, powerfully so. Response to the art component was mixed; for some student-patient pairs, the art the patient chose brought new insights into patient understanding. For other pairs, the technology and/or image library created

• barriers. Interviews with patients revealed generally positive response, although some patients did not recall the activity. Several

patients asked their students to be their primary care provider. Comparison of audiotaped patient stories to the corresponding written narratives penned by students is ongoing to assess students' narrative competence. DISCUSSION / REFLECTION / LESSONS LEARNED: A brief, experiential narrative medicine curriculum was feasible and acceptable to students and patients. From a time and resource perspective, the curriculum took two hours of faculty time per month. Both patients and students appear to have yielded benefits - for patients, being attended to and being heard; for students, gaining deeper appreciation of the human side of medicine, of knowing patients better. The impact on the student-patient relationship was

• ften positive, at times powerfully so. Ongoing focus groups helped to make continual improvements to the curriculum, including the

retirement of the iPad art component and refinement of how the patient activity was structured to maximize comfort of both students and patients.

MOBILE-CENTERED SPACED LEARNING: A NOVEL APPROACH TO GRADUATE MEDICAL EDUCATION IN THE DIGITAL ERA Robert J. Walter; Anantha K. Mallia; Ali Massoumi; Joshua Hartzell. Walter Reed National Military Medical Center, Bethesda, MD. (Tracking ID #1938605) NEEDS AND OBJECTIVES: Developing an optimal learning environment and curriculum in graduate medical education can be challenging, particularly in light of progressive work hour restrictions limiting the feasibility of traditional academic didactics that compete with patient care duties. Emerging literature has demonstrated the effectiveness of "spaced learning" within medical education as a viable adjunct to conventional pedagogy. This method of learning has been found to improve knowledge acquisition, increase long-term knowledge retention, and improve a learner's ability to accurately self-assess their knowledge. However, spaced learning shares the same limitations as many other learning models with regard to accessibility, ease of use, and learner acceptance/adoption. Additionally, no studies to date have evaluated the usefulness of a mobile-based spaced learning curriculum within an internal medicine residency program. As such, the goal of our study was to assess the effectiveness and acceptance of a mobile-based spaced learning curriculum within a graduate medical education environment. SETTING AND PARTICIPANTS: Utilizing a mobile-based application, Qstream, that incorporates an adaptive spaced learning platform, we created an adjunct curriculum to supplement our existing academic didactics for housestaff during their critical care rotation at Walter Reed National Military Medical Center (WRNMMC). The curriculum consisted of 100 high-yield core knowledge questions within the area of critical care medicine. Each learner enrolled in the module received an email and/or RSS feed prompt daily to their mobile device of preference (smartphone, tablet) with 5 questions. Upon answering each question, a brief explanation of clinical reasoning was provided and the learner was anonymously shown their performance in comparison to their cohort. Questions that were answered incorrectly were asked again within 7 days until answered correctly twice, whereas questions initially answered correctly were asked again within 13 days. Questions answered correctly twice in succession (mastery of material) were retired. DESCRIPTION: We conducted a prospective, randomized, controlled study of 30 interns and residents rotating on the medical and surgical intensive care units at WRNMMC. All participants had their baseline knowledge assessed by a 25 question pre-test based upon the core knowledge topics within our spaced learning module. 16 learners were randomized to participate in the spaced learning curriculum (in addition to traditional didactics) and were enrolled within the module. 14 learners were randomized to the traditional academic didactics alone. Block randomization was used to minimize cross-communication between learners. All participants completed a 25 question post-test at the conclusion of their 4 week rotation. Learners who were randomized to participate in the spaced learning module answered an additional 10 questions regarding the perceived effectiveness and usability of the module. EVALUATION: Our cohort consisted of 18 PGY-1's (60%), 5 PGY-2's (17%), 6 PGY-3's (20%), and 1 PGY-4 (3%). Baseline characteristics (PGY level) were similar between randomized groups. Performance on the pre-test examination was comparable (61% +/- 18% vs 68% +/- 17%, p = 0.26). Post-test performance was superior within the spaced learning group (86% +/- 11% vs 71% +/- 13%, p = 0.002) as was the percent improvement from pre-test to post-test (18% +/- 12% vs 7% +/- 13%, p = 0.03). The spaced learning module received favorable reviews both in terms of overall educational benefit (average rating 4.44 on a 5-point Likert scale) and as an adjunct to traditional didactics and bedside teaching (average rating 3.56 on a 4-point Likert scale). The average time to complete the 5 daily questions was 4.25 minutes. The overall difficulty and number of questions administered daily were designated as "appropriate" by 81% and 87% of learners, respectively. DISCUSSION / REFLECTION / LESSONS LEARNED: In our cohort, we observed a statistically significant improvement in

• verall post-test performance within the spaced learning group. Furthermore, acceptance and perceived usefulness of the mobile-based

platform was high among learners. This novel approach to traditional didactics may be useful as a means to supplement existing

• curricula. Additional measurements, such as in-service and ABIM exam scores, gathered from larger, multi-center trials are needed to

confirm the overall utility of a spaced learning curriculum as an educational adjunct within graduate medical education.

HOME VISITS FOR MEDICALLY COMPLEX PATIENTS IN A TERTIARY ACADEMIC MEDICINE CLINIC AS A TOOL FOR RESIDENT EDUCATION AND READMISSION PREVENTION Duncan Vincent; Paul Chelminski. University of North Carolina Hospitals, Chapel HIll, NC. (Tracking ID #1897469) NEEDS AND OBJECTIVES: Home visits are a much-valued but under-utilized way to provide care. Historically, providers have

• ffered these in urban settings for logistical ease. We developed a multi-modal educational and clinical initiative with three principle

aims: provide structured care outside of a standard clinic visit; provide rapid post-hospitalization follow-up for a subset of patients to decrease readmissions; and integrate resident physicians into this process as practitioners and learners. SETTING AND PARTICIPANTS: The Charles Sanders Clinical Scholars Grant was created to promote personalized and humanistic medical care at the University of North Carolina at Chapel Hill as part of the educational mission. The first recipient of the award (an attending clinician educator) developed a home visit program that integrated resident physicians. Residents on ambulatory blocks accompanied the attending on home visits. The population consisted of established patients in the UNC Internal Medicine Clinic, which is a blended faculty and resident practice. Shortly after its inception, the program expanded to make the resident the lead physician in an initiative to decrease readmissions of recently discharged, high risk patients. DESCRIPTION: Home visits were arranged by a multidisciplinary team including a social worker. They were conducted by the attending physician and usually one resident physician (sometimes a pharmacy resident). Residents were the lead physician for hospital follow-up visits. Medication reconciliation occurred at each visit with direct inspection of the patient's pill bottles. Patients received counseling about medication adjustment, new prescriptions, laboratory, or imaging studies. Almost all visits were billed. After the visit, residents wrote a brief reflection of the experience. A master map was created of all the home visit locations. EVALUATION: To date, there have been 62 unique home visits and 16 subsequent follow-up visits. Mean age of patients was seventy years (range 29 - 95 years). Fifty-two percent were male. Mean and median distances traveled round trip were 20 and 13 miles respectively that spanned six counties (range 0 - 95 miles). Average length of the visit was thirty minutes (range 10 - 75 minutes). Seventy-six percent of patients were insured with Medicare, 16% with private insurance, 2% with Medicaid, and 6.5% were

• uninsured. Seventy-seven percent of patients were Caucasian and 21% were African American. The patients were medically complex:

56% had hypertension, 22% had congestive heart failure, 29% had coronary artery disease, 13% had atrial fibrillation, 17% had COPD, 16% had diabetes, and 13% had depression. Only ten visits (16%) were conducted outside of usual clinic hours. Twenty-four visits (39%) were urgent care visits for acute complaints. Ten percent of home visits were conducted for palliative care and hospice

• monitoring. An internal medicine resident was present for 75% of the visits. There was a new prescription at 30% of all visits. Forty-

three percent were hospital follow-up visits occurring within fourteen days of discharge. Seven percent of these patients were readmitted to the hospital within thirty days of discharge. By comparison, our general medicine clinic population has a 19% re- admission rate within thirty days. A gift was offered at 17% of visits. In 20% of the households, the television was on in the examining space. A sampling of learner reflections after the visits revealed an enhanced appreciation for the social and environmental dimensions of care. DISCUSSION / REFLECTION / LESSONS LEARNED: Home visits span a broad spectrum of patient encounters, from acute to chronic to palliative care. Interestingly, the median visit duration was not extended when compared to a standard clinic visit. We have demonstrated the feasibility of home visits in a non-urban teaching medical center. They circumvent barriers to prompt, patient- centered care that prevail in the traditional clinic visit. They can potentially prevent hospital readmissions and decrease resource

• utilization. Integrating learners into home visits promotes humanism in medicine. Resident-composed reflections emphasize the

reciprocal nature of care between patients and providers in this non-time-pressured and non-regimented setting.