Validation of a tool for summative assessment of endoscopy - - PowerPoint PPT Presentation

Validation of a tool for summative assessment of endoscopy performance

Christen K. Dilly, Zach Morgan, Smitha Marri, Nabil Fayad

Background

• Used at the end of each

month of training

• Completed by 1-2 faculty

members per fellow

• Construct validity:

developed using ASGE recommendations1

1. ASGE Training Committee. Principles of training in GI endoscopy. Gastrointestinal Endoscopy. 2012; 75(2):231-235.

Results - Reliability

• Internal consistency -

Cronbach’s alpha

– Total = 0.957 – Technical skills = 0.967 – Non-technical skills = 0.928.

• Inter-rater

reliability – Kappa

– Categorical Question Factor Agreement

Technical Non- technical Kappa

1 – Manipulates scope 1.015

• 0.084

0.259** 2 – Mucosal eval 0.981

• 0.031

0.316** 3 – Therapeutic interventions 0.928 0.040 0.254** 4 – Procedure indications 0.621 0.336 0.091 5 – Recognizes, interprets, manages findings 0.815 0.171 0.065 6 – Anticoagulant/ antiplatelet/antibiotic 0.421 0.509 0.048 7 – Teamwork

• 0.008

0.860

• 0.010

8 – Feedback 0.059 0.739 0.044 9 – Quality benchmarks 0.952

• 0.003

0.471** 10 – Informed consent/safety

• 0.015

0.890 0.026 11 – Responsibility 0.020 0.863

• 0.014

12 - Documentation 0.064 0.818

• 0.010

**= Significant to <0.001

Results - validity

• Pearson correlations

(relationship of procedure numbers to assessment scores):

– Technical skills: 0.606 – Nontechnical skills: 0.249

• Response process – cognitive

interviews

Conclusions:

• Fair to moderate evidence for reliability and validity
• Problems with the instrument likely reflect problems with milestones-based

assessment

• How will this compare to EPAs?