Competence fair to long standing doctors? L Mehdizadeh, A Sturrock, - - PowerPoint PPT Presentation

Are the GMC’s Tests of Competence fair to long standing doctors?

L Mehdizadeh, A Sturrock, J Dacre

Presented by Dr Leila Mehdizadeh Research Associate UCL Medical School l.mehdizadeh@ucl.ac.uk

Academic Centre for Medical Education

• Research department of

UCL Medical School

• Remit for educational

research

• Contract with GMC FtP until

2019

• Design and delivery of the

Tests of Competence (ToC)

GMC performance assessment

On site

• Peer led work based assessment

Off site

• Test of competence
• Knowledge test (SBA’s)
• Practical skills test (OSCE)

Tests of competence

• The ToCs are not pass/fail

examinations

• The results will be

considered with all other evidence when coming to a judgement about the doctor’s FtP

• The evidence may however

be subject to challenge by defence lawyers

• Doctors under investigation
• Volunteer doctors
• pilot the ToC material
• reference group for comparison

Study participants

Study rationale

• Average age of doctor undergoing performance

assessment was 55

• High proportion of doctors under investigation

known to be overseas trained

• SBA and OSCE are newer test formats that some

doctors may be unfamiliar with

Qualification year and test preparation

• Evidence that increase in age and time elapsed

from qualification year is associated with decline in clinical performance (Chaudhry et al 2005; Duclose et

al 2012; Caulford 1994).

• Test preparation and test familiarity is linked to

improved clinical performance (Goldberg and Pedulla

2002, Bangert-Drowns, & Kulik, 1984).

Does year of qualification predict ToC performance

• in doctors undergoing FtP investigation?
• in doctors with no known concerns?
• when controlling for gender, ethnicity and

region of PMQ in both groups of doctors?

Research questions

Methods

Retrospective cohort design Compared 95 GPs under FtP investigation vs 376 volunteer doctors Outcomes

• knowledge test score (total 120)
• OSCE overall score (total 480)

Results: demographics

FtP group (n = 95) Volunteer group (n = 376) Total on GP register (n = 63,778)

Gender Male 79 (83%) 169 (45%) 32, 719 (51%) Female 16 (17%) 207 (55%) 31, 059 (49%) PMQ region UK 34 (36%) 324 (86%) 49, 198 (77%) EEA 7 (7%) 4 (1%) 3, 998 (6%) International 54 (57%) 48 (13%) 10, 582 (17%) Ethnicity White 27 (28%) 220 (59%) Black 17 (18%) 16 (4%) Asian 39 (41%) 112 (30%) Mixed 8 (2%) Other 12 (13%) 18 (5%) Not stated 2 (<1%) Year of qualification Before 1993 (early) 85 (89%) 69 (18%) 1993-2005 10 (11%) 146 (39%) 2006-2011 (recent) 161 (43%)

Knowledge test performance: volunteer group

Doctors under investigation

OSCE performance: volunteer group

Doctors under investigation

Multivariable linear regression

Doctors under investigation Volunteer doctors Performed worse than volunteers on all outcomes Performed better than FtPs

• n all outcomes

Earlier graduates performed worse than more recent graduates on KT and OSCE overall Recent graduates performed worse than earlier graduates

• n KT, but on OSCE those

who qualified after 2005 performed best. Doctors under investigation who qualified prior to 1996 performed significantly worse on KT and OSCE

• verall.

Qualification year was more strongly associated with ToC performance in FtP doctors than volunteers

Summary of effect of qualification year when controlling for gender, ethnicity and qualification region

Discussion

• Relationship between qualification year and ToC

performance differs between two groups

• If ToC format was unfair to older doctors we would

expect to see similar pattern in volunteers

• Why does performance of FtP doctors decline?
• Why do recently qualified volunteers do worse in

KT and better in OSCE?

Strengths and limitations

Large sample, few studies explore effect of qualification year on high stakes exam Demographics differ between two groups

Most FtP qualified prior to 1992 vs volunteers from 2006+

Volunteers that graduated before 1996 performed better on all outcomes Data only for general practice

• Yes in doctors undergoing investigation
• Less so in volunteer doctors

Does qualification year predict ToC performance?

Take home messages

ToC are fair irrespective of when doctors qualified

Need to extend study with:

• more doctors under investigation who are recent

graduates

• more volunteer doctors who are early graduates
• data from range of clinical specialties
• control for doctors’ age and years in practice

Thank you for your attention

l.mehdizadeh@ucl.ac.uk