Development of viva voce exam to assess clinical reasoning in the - - PowerPoint PPT Presentation

Development of viva voce exam to assess clinical reasoning in the paraclinical phase of a BVSc program.

Helen Owen, Senior Lecturer in Veterinary Pathology and Jus:ne Gibson, Senior Lecturer in Veterinary Bacteriology and Mycology Dr Tamsin Barnes Associate Professor Rowland Cobbold Dr Katrina GarreG Dr Steven Kopp Associate Professor Joanne Meers Dr Dan Schull Associate Professor Jenny Seddon Dr Frances Shapter Associate Professor Rebecca Traub

Background on Program

• The BVSc at UQ is an undergraduate 5 year program
• First 3 years consist predominantly of preclinical sciences with the

fourth and fi?h years having more of a clinical focus.

• Major courses in third year
• a year-long course on pathophysiology – stereotypical ways

which Fssues respond to injury, prototype diseases

• clinical signs, clinical exam and diagnosFc approach to body

system

• course on infecFous disease (parasitology, virology, bacteriology

and mycology).

Development of viva voce exam

• In 2012, we were tasked with developing a viva voce oral exam as a program level

“hurdle” assessment at the end of third year. ObjecFves, to assess:

• applicaFon of theory to case scenarios
• diagnosFc clinical reasoning
• integraFon of material from both the pathophysiology and infecFous diseases

courses

• professionalism and oral communicaFon skills
• Exposure to oral exam format before 5th year viva exams

Advantages of oral format

• Good for assessing problem-solving abiliFes.
• Can provide insight into students’ cogniFve processes.
• It allows probing of the depth and extent of students’ knowledge with

follow-up quesFons

• Unclear or ambiguous quesFons can be re-expressed or immediately

clarified

• Students can be guided back onto the right track if they stray

Advantages of oral format

• Allows judgements about students’ interpersonal competence -

confidence, self-awareness and aspects of professionalism.

• SimulaFon of clinical pracFce – students need to “think on their feet”
• It improves the quality of student learning – students have been

reported to strive for thoroughness in understanding in response to the unpredictability and to avoid embarrassing themselves in front of the examiner

• It suits some students

Disadvantages of oral format

• Generates a lot of anxiety in students and examiners
• Time consuming and requires a lot of energy
• Lack of anonymity and potenFal for bias – unconscious bias based on

charismaFc personality, well dressed, being arFculate

• Reliability - Inter-case - different students get different cases to prevent

sharing of informaFon.

• Inter-rater
• Intra-rater reliability

Features of the initial exam

• Formulaic series of quesFons based on the diagnosFc process
• Provided with image and some informaFon

– history and clinical exam findings

• body system(s)
• possible pathological processes (high weighFng) – using

pathophysiology for clinical reasoning

• differenFal diagnoses (low weighFng!)
• diagnosFc plan (high weighFng)
• Interpret results, treatment, control, prognosis
• professionalism

Initial exam

• All students received one case
• Students who failed iniFal exam were given a resit
• 10 minutes perusal, 20 minutes oral
• 2 examiners per student - increase reliability – consensus marks used
• Rubric and model answers - increase reliability

Changes over the years based on examiner and student feedback

• 2 cases (usually one large animal, one small) – increases reliability
• More examiner training
• Removed history and clinical exam secFons
• 2-stage pathophysiology
• Adjusted rubric - more emphasis on systemaFc approach for

pathogenesis and diagnosFc tesFng, forced to prioriFse DDxs

• Year-long preparaFon and provision of formaFve feedback.

You are presented with a 1 year old en3re male Domes3c Short Hair cat. The owner no3ced that the cat is off its food and ‘quieter’ than usual, this has been the case for about a week.

Exam format – written section (30 minutes)

Part A. Descrip:on of case, presen:ng signs (images/ videos)

Exam format – written section

• Part B. Outline the pathological processes which may be causing the

presentaFon

• Students sFll rote learning pathogenesis for parFcular clinical

presentaFon without emphasising/ prioriFsing based on case parFculars

• Or students using pacern recogniFon and being able to describe one

specific pathogenesis but not being able to suggest alternaFves

• Break down into two steps, in the first step, they need to demonstrate

a systemaFc knowledge structure that includes all possible pathogeneses

• Second step, provided with addiFonal informaFon about the case and

need to use this to prioriFse DDxs

• Encourage students to consider all possible pathogeneses for a broad clinical sign at this

point

• Must demonstrate structure, systemaFc approach, encouraged to use schemes

Scheme-inductive knowledge structures

Harasym et al. 2008 Kaohsiung J Med Sci, 24:7; 341-355.

Research suggests that expert problem solvers are disFnguished by the way in they organise and understand their knowledge

Scheme-inductive knowledge structures

• Mandin et al, University of Calgary Faculty of Medicine – “Clinical

PresentaFon” curriculum

• “The “hypotheFco-deducFve” strategy tradiFonally used for PBL

should be replaced by scheme-driven search strategies so that students develop a more organised and logical approach to problem-solving” Schemes have mulFple purposes:

• Provide systemaFc approach to decrease the risk
• f omidng DDxs

• way of organising knowledge for more efficient understanding, serve as a scaffold

for new informaFon

The history, physical examinaFon and laboratory invesFgaFon are all driven by the derived scheme

Paraclinical schemes

• Probably different content to clinical schemes, more reliance on

principles of pathophysiology, less signalment etc? Nodes can consist of:

• locaFon e.g. body systems - cardiovascular versus respiratory tract;

within body systems - upper versus lower respiratory tract;

• a form of general pathological process in the Fssue e.g. myocardiFs,

hepaFc necrosis, pulmonary oedema

• Specific causaFve agents (prototypic)
• Some presentaFons a bit different e.g. anaemia, jaundice
• Lots of schemes provided to the students and they are encouraged to

improve on them

Mnemonics as a system?

• e.g. DAMNITV - DegeneraFve, Anomalous, Metabolic etc
• Students o?en prefer this approach
• trying to not be too prescripFve, flexibility allowing for different thought

processes, clinical presentaFon Don’t work too well in our hands ….

• Doesn’t require explicit idenFficaFon of body systems
• Students o?en don’t seem to understand pathogenesis
• Doesn’t provide structure for systemaFc diagnosFc approach
• More appropriate as a memory jogger in clinical years a?er students have

demonstrated pathophysiology knowledge?

• Useful as part of a mulFple strategy approach?

You will be provided with addiFonal history or clinical features

• You perform a PCV and there is marked anaemia.
• IniFally we assume that good students would prioriFse DDxs based on all

the features of the case but this rarely happened Part C. Give a problem defini:on

• Subacute inappetence, depression, jaundice and anaemia.

Exam format – written section

Exam format – oral section

Part D. Body system(s)?

• haematopoieFc
• Part E. Use the addiFonal informaFon to idenFfy most likely pathological

process(es), describe pathogenesis and provide aeFological examples if appropriate

Part F. List 5 differenFal diagnoses Part G. DiagnosFc plan

• IniFally found that students would just test and Fck off each of their

differenFals

• Encourage systemaFc approach, use of scheme to guide, most

informaFon from fewest tests

• Encourage use of diagnosFcs to guide treatment
• Students do relaFvely poorly in this secFon

Exam format – oral section

Exam format – oral section

Provide diagnosis, prognosis, treatment plan, control This secFon also poorly done in general

Exam format – oral section

• Part I – professionalism – use of professional language, professional

behaviour and dress.

Outcome of viva exams

• This is a must-pass hurdle, a larger number of students fail this compared to failing
• n basis of theory exams (~5-9/120 versus ~2-3)
• O?en about 10 students on pass/ fail line, reluctant to let them go into 4th year –

what should our expectaFons be for these novice diagnosFcians?

• Most students who fail viva are weak in theory also but not always

Preparation for viva exams

• 6 face-to-face tutorials dispersed through the year
• 5th year students act as tutors for small groups
• 3rd years examining each other in pairs
• Include all steps in process, even at the beginning
• 3 on-line case-based tutorials where generic feedback given but students can elect to

have personal feedback

• “diagnosFc work-shop”
• Viva “hint-sheet”

Lessons we have learnt

• We took a lot fore granted iniFally: “good students should automaFcally demonstrate a

systemaFc/ structured approach, prioriFse DDxs based on parFculars of a case …”

• Need to be completely explicit in expectaFons and specific for each step in process which

means having a good understanding of them in your own mind

• Need to structure rubric so that it is not possible for a student to get good marks without

a systemaFc approach and being able to prioriFse

• Need to structure preparaFon material to encourage good habits
• Encourage pracFce of verbal communicaFon as this is a very different skill to wricen

Conclusions

Oral format benefits:

• Being able to guide students onto the right track, test “thinking on feet”, probe

understanding

• Fear is a powerful moFvator for embracing new learning approaches

Oral format limitaFons:

• VariaFon in cases – lack of standardisaFon?
• Inter-rater reliability
• Massive amount of Fme and energy, difficult to engage examiners
• InternaFonal students disadvantaged by language, culturally ingrained behaviour?
• Really good format for assessing diagnosFc clinical reasoning

(oral communicaFon skills, professionalism) but does good

• utweigh the bad?

Future directions

• Reliability – variaFon in marks across cases, examiners, secFons of exam
• InvesFgate whether preparaFon and taking these exams has a lasFng influence on

students’ study habits and approach to cases

• Developing more efficient means of providing formaFve assessment
• Becer integraFon with the clinical years of the program – evolving nature of

clinical reasoning strategies, rolling out of series of verFcally integrated on-line tutorials

• Obtain becer understanding of approaches of weak students so as to becer assist

their preparaFon

References

• Davis, M., Karunathilake, I. 2005. The place of oral examinaFons in today’s assessment
• systems. Medical Teacher, 27; 4: 294-297.
• Harasym et al. 2008 Current Trends in Developing Medical Students' CriFcal Thinking
• AbiliFes. Kaohsiung J Med Sci, 24:7; 341-355.
• Joughin, 2010. A short guide to oral assessment. Leeds Metropolitan Press in AssociaFon

with University of Wollongong.

• Mandin, H., Harasym, P., Eagle, C. and Watanabe, M. 1995. Developing a “clinical

presentaFon” curriculum at the University of Calgary. Academic Medicine. 70; 3: 186-193.

• Mandin, H., Jones, A., Woloschuk, W. and Harasym, P. 1997. Helping students learn to

think like experts when solving clinical problems. Academic Medicine, 72; 3: 173-179.