RESIDENCY-IN-TRAINING EVALUATION (RITE) ANNUAL WRITTEN EXAMINATION - - PowerPoint PPT Presentation

RESIDENCY-IN-TRAINING EVALUATION (RITE) ANNUAL WRITTEN EXAMINATION & INTERNAL MEDICINE ADMISSIONS TEST(IMAT) 2013 Lenora Fernandez, MD, FPCP

OVERVIEW on RITE

• RITE/REE held yearly 1987-2003 then stopped

when validity & reliability in doubt

• Voluntary on-line format 2009-2011: very few

takers

• Annual written exam format resumed 2013 =

OCTOBER 13, 2013, Sunday

OBJECTIVES of RITE

General Objective:

• To determine the cognitive skills of internal medicine residents in all

PCP-accredited training programs. Specific Objectives:

– To compare the cognitive skills of IM residents among all accredited internal medicine training programs in the Philippines. – To compare the cognitive skills between 2nd and 3rd year IM residents within the institution and all other institutions. – To determine the areas of strengths and weaknesses of the examinees and serve as guide in maintaining and improving their training programs. – To provide a tool in setting standards for accreditation of IM training programs to the Committee on Accreditation in (future) – To determine if this examination can predict the performance of the residents in the PSBIM.

OVERVIEW on RITE

• Multidisciplinary committee
• Tried to simulate the PSBIM
• 200 items: 175 MCQs, 25 Modified True or False
• Chapters hosted and conducted the RITE

Cardiology Pulmonary Medicine Endocrinology Gastroenterology Nephrology Infectious Diseases Hematology Allergology-Immunology Rheumatology Medical Oncology Neurology Dermatology Ethics

Recall - 30% Based on PCP Glossary of Top Comprehension – 30% Diseases Analysis – 40%

23 RITE TEST SITES

Fatima College of Medicine, Valenzuela City (CAMANAVA) Laoag City Dagupan City Baguio City Tuguegarao City Legaspi City Lipa City Tacloban City Davao City Cagayan de Oro City Zamboanga City Quezon City Manila City Marikina City Makati City (MATAPAT) Las Pinas City (PAMUNLAS) Pasay City Pasig City (PASJMAN) Angeles City, Pampanga Cebu City Tagbilaran City Iloilo City Bacolod City

RITE DEMOGRAPHICS

Total number of examinees 1264, 94 training institutions Year Level 2 examinees N = 476 Mean score = 102.38/200 Year Level 3 and 4 examinees (no.) 469 Mean score = 106.1/200 Post-residency examinees (no.) 319 Mean score = 94.7 Set minimum passing level*: Year Level 2 Year Level 3, 4 and post-residency level 86 out of 200 107 out of 200 RITE exam analysis: Range: Mean: Median: Mode:

Reliability index:

65.2-153.8 101.8 100.8 108.2

0.83 (vs 0.63-0.78 in previous)

* MPL pre-determined: YL 2 – 10% lower than YL3/4 (71% of 200 items); MPL lowered by 1 SD after results collated

RITE TEST INDICES

Difficulty index:

10% very easy 25% easy 15% moderate 30% difficult 20% very difficult < 1% very poor 50% poor 30% good 20% excellent

Discrimination Index:

MEAN RITE SCORES IN 3 GROUPS

80 85 90 95 100 105 110

2nd year 3-4th year Postgraduate 102.4 + 14.6 106.7 + 15.3 94.7 + 11.67 N = 476 469 319 Mean RITE Scores (over 200 items) p < 0.001 p < 0.001

PERFORMANCE IN DIFFERENT TOPICS OF THE 3 GROUPS

0.00% 20.00% 40.00% 60.00% 80.00% YL2 YL3-4 PGRAD % mean score in topic

TOP 25 TRAINING INSTITUTIONS IN PERCENTILE RANK, RITE

UP-PGH National Kidney & Transplant Institute University of Santo Tomas Premiere Medical Center Cardinal Santos Medical Center Zamboanga City Medical Center Ospital ng Maynila Medical Center FEU-NRMF Medical Center Makati Medical Center Davao Doctors Hospital Western Mindanao Medical Center St Lukes Medical Center UERM Memorial Medical Center SLU-Hospital of the Sacred Heart Chong Hua Hospital West Visayas State University Medical Center University of Perpetual Help Dr. Jose G. Tamayo Medical Center-Binan, Laguna Southern Philippines Medical Center Divine Word Hospital Chinese General Hospital and Medical Center

• St. Paul's Hospital

AFP Medical Center Manila Adventist Medical Center Mary Mediatrix Medical Center

• Dr. Pablo O. Torre Memorial Hospital

YEAR LEVEL 3 RITE - PSBIM

20 40 60 80 100 120 Passed PSBIM Failed PSBIM Did not take PSBIM

94.45 + 9.38 114.41 + 12.84 98.79 + 13.64

p < 0.001

Mean RITE Scores (over 200 items)

N = 232 51 186

POSTGRADUATE EXAMINEES: RITE - PSBIM

84 86 88 90 92 94 96 98 100 102 104

Passed PSBIM Failed PSBIM Did not take PSBIM

93.5 + 10.16 102.7 + 10.4 90.75 + 10.4

p < 0.001

Mean RITE Scores (over 200 items)

N = 79 116 124

FEEDBACK OF EXAMINERS/PROCTORS

• Clear instructions? YES- 100%
• Enough time to prepare for organization of RITE? YES – 96.1%
• Conflicts in the instructions given? NONE 78.7%
• Do you agree to the involvement of PCP chapters in the conduct of

RITE? YES – 99%

• Will you be willing to serve again as proctor or organizer for the

RITE in the coming years? YES- 98%

• Further comments:

– Role of RITE in regular evaluation of residents? – Notify earlier, orientation meeting, minimize conflicts in instructions – Separate exam for 2nd and 3rd years – Include first years – Chairmen to encourage post-residency members who want to take PSBIM – Compensation to staff?

FEEDBACK OF EXAMINEES

• Was RITE given at the right time of your training? YES 86.7%
• For residents-in-training, test questions relevant and appropriate to your

level of training? YES 98.3%

• For post-graduate examinees, test questions serve as a “good review” for

your coming PSBIM exam? YES 98.4%

• Enough time allotted to answer all questions? YES 96.2%
• Enough time to review for the exam? YES 55.33%
• Details provided beforehand sufficient? YES 87.6%
• Venue conducive to answering exam well? YES 91.2%
• Proctors helpful in clarifying your questions during the exam? YES 99.7%
• Will you recommend that RITE be regularly conducted on an annual basis?

YES 95.1%

• Would you prefer RITE to be conducted on-line? YES 37.4%

FEEDBACK OF EXAMINEES

• Must include first years, more often
• Venue issues (temp, announcement, chairs/desk,

comfort room, crowding)

• Questions (different from PSBIM, relevance)
• No questions allowed
• Coverage/blueprint not announced earlier
• Must be paired with a post-exam review

CONCLUSIONS ON RITE ANNUAL WRITTEN EXAM:

• The RITE 2013 met most of specific objectives it aimed for:

– compare the cognitive skills of IM residents among all IM training programs. – DONE – compare cognitive skills between 2nd and 3rd year IM residents within the institution and all other institutions. - DONE – determine the areas of strengths and weaknesses of the examinees and serve as guide in maintaining and improving their training programs. – DONE – determine if this examination can predict the performance of the residents in the PSBIM.  Based on the results, it seems it can predict the performance in the PSBIM.

• Reliability index high at 82.9% + opinion of majority among

the examiners and examinees to continue with RITE.

• The RITE is recommended to be continued.

RECOMMENDATIONS

• Reformulation of RITE Committee examiner composition .
• More training in test construction.
• Next date: October 12, 2014, Sunday.
• More test sites nearer to examinees to be manned by Chapters.
• Better and earlier coordination with the PCP chapters.
• Test blueprint given today:

– ALL MCQ’s – 250 items – include questions on research and principles on critical appraisal of literature – Integrative questions across different topics;subdiv to ambulatory, emergency, or critical care dimensions.

• Scantron machine already available at PCP
• More efficient system of dissemination and recording/collating
• Data to be continually analyzed
• Discussion on bearing on internal evaluation of resident will be
• pened?

INTERNAL MEDICINE ADMISSIONS TEST

Objectives of IMAT:

• Provide a uniform assessment standard for

entrance into IM training programs

• Create a profile of aspiring entrants
• Determine baseline knowledge of individual

aspiring entrants in comparison to the general pool of aspirants through normative statistical analysis.

OVERVIEW & CONDUCT

• 100-item MCQ written exam
• Questions to cover only pathophysiologic basis of

disease, general diagnostic and therapeutic principles of general disease categories (cardio, pulmo, endo, IDS, nephro, etc.)

• No MPL set
• Given to TI’s with answer key for immediate

utilization in their screening process; can administer at any time

CORRELATION OF IMAT WITH MEDICAL BOARD EXAM RATING

75 80 85 90 Overall Medical Boards Rating (%) 20 40 60 80 100 IMAT Score (raw over 100 items)

Pearson’s correlation coefficient = 0.687 N = 266 evaluable results

CHALLENGES:

• Lack of security of the exam:

– Leakages unavoidable since administered at different dates

• Provision for applicants to different

institutions

• Relevance in screening process?

– What is/are main factor/s in accepting new residents into training program?

RECOMMENDATION:

• CONSENSUS decision among Regents and

Training Institutions to STANDARDIZE SCREENING PROCESS for IM training applicants with a written standardized examination as just ONE of the screening tools

– Uniformity in time of screening can then be ensured – Security and confidentiality then of IMAT can be ensured

Thank you!

• Dr. Lenora Fernandez

Chair

• Dr. Adrian Rabe

Assistant Chair

• Dr. RontgeneSolante

IDS

• Dr. Sonia Salamat

IDS

• Dr. Beatriz Concepcion

Nephrology

• Dr. Elaine Cunanan

Endocrinology

• Dr. Heizel Reyes

Rheumatology/Ethics

• Dr. Norman Maghuyop

Pulmonary Medicine

• Dr. Jesus Relos

Hematology

• Dr. Dennis Sacdalan

Oncology

• Dr. Richard Tiongco

Cardiology

• Dr. Eternity Labio

Gastroenterology

• Dr. Cherie Cervantes

Immunology-Allergology

Adviser: Regent Nenita Collantes, MD, FPCP