Enlighten
Research publications by members of the University of Glasgow
home > services > Enlighten

The reliability of workplace-based assessment in postgraduate medical education and training: a national evaluation in general practice in the United Kingdom

Murphy, D.J., Bruce, D.A., Mercer, S.W., and Eva, K.W. (2009) The reliability of workplace-based assessment in postgraduate medical education and training: a national evaluation in general practice in the United Kingdom. Advances in Health Sciences Education, 14 (2). pp. 219-232. ISSN 1382-4996 (doi:10.1007/s10459-008-9104-8)

Full text not currently available from Enlighten.

Publisher's URL: http://dx.doi.org/10.1007/s10459-008-9104-8

Abstract

To investigate the reliability and feasibility of six potential workplace-based assessment methods in general practice training: criterion audit, multi-source feedback from clinical and non-clinical colleagues, patient feedback (the CARE Measure), referral letters, significant event analysis, and video analysis of consultations. Performance of GP registrars (trainees) was evaluated with each tool to assess the reliabilities of the tools and feasibility, given raters and number of assessments needed. Participant experience of process determined by questionnaire. 171 GP registrars and their trainers, drawn from nine deaneries (representing all four countries in the UK), participated. The ability of each tool to differentiate between doctors (reliability) was assessed using generalisability theory. Decision studies were then conducted to determine the number of observations required to achieve an acceptably high reliability for "high-stakes assessment" using each instrument. Finally, descriptive statistics were used to summarise participants' ratings of their experience using these tools. Multi-source feedback from colleagues and patient feedback on consultations emerged as the two methods most likely to offer a reliable and feasible opinion of workplace performance. Reliability co-efficients of 0.8 were attainable with 41 CARE Measure patient questionnaires and six clinical and/or five non-clinical colleagues per doctor when assessed on two occasions. For the other four methods tested, 10 or more assessors were required per doctor in order to achieve a reliable assessment, making the feasibility of their use in high-stakes assessment extremely low. Participant feedback did not raise any major concerns regarding the acceptability, feasibility, or educational impact of the tools. The combination of patient and colleague views of doctors' performance, coupled with reliable competence measures, may offer a suitable evidence-base on which to monitor progress and completion of doctors' training in general practice.

Item Type:Article
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Mercer, Prof Stewart
Authors: Murphy, D.J., Bruce, D.A., Mercer, S.W., and Eva, K.W.
College/School:College of Medical Veterinary and Life Sciences > Institute of Health and Wellbeing > General Practice and Primary Care
Journal Name:Advances in Health Sciences Education
ISSN:1382-4996

University Staff: Request a correction | Enlighten Editors: Update this record