Adaptive comparative judgment: a tool to support students' assessment literacy

Rhind, S. M., Hughes, K. J., Yool, D. , Shaw, D., Kerr, W. and Reed, N. (2017) Adaptive comparative judgment: a tool to support students' assessment literacy. Journal of Veterinary Medical Education, 44(4), pp. 686-691. (doi: 10.3138/jvme.0616-113R) (PMID:28581915)

Abstract

Comparative judgment in assessment is a process whereby repeated comparison of two items (e.g., assessment answers) can allow an accurate ranking of all the submissions to be achieved. In adaptive comparative judgment (ACJ), technology is used to automate the process and present pairs of pieces of work over iterative cycles. An online ACJ system was used to present students with work prepared by a previous cohort at the same stage of their studies. Objective marks given to the work by experienced faculty were compared to the rankings given to the work by a cohort of veterinary students (n=154). Each student was required to review and judge 20 answers provided by the previous cohort to a free-text short answer question. The time that students spent on the judgment tasks was recorded, and students were asked to reflect on their experiences after engaging with the task. There was a strong positive correlation between student ranking and faculty marking. A weak positive correlation was found between the time students spent on the judgments and their performance on the part of their own examination that contained questions in the same format. Slightly less than half of the students agreed that the exercise was a good use of their time, but 78% agreed that they had learned from the process. Qualitative data highlighted different levels of benefit from the simplest aspect of learning more about the topic to an appreciation of the more generic lessons to be learned.

• Item Type: Articles
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Yool, Professor Donald
• Authors: Rhind, S. M., Hughes, K. J., Yool, D., Shaw, D., Kerr, W., and Reed, N.
• College/School: College of Medical Veterinary and Life Sciences > School of Biodiversity, One Health & Veterinary Medicine
• Journal Name: Journal of Veterinary Medical Education
• Publisher: University of Toronto Press
• ISSN: 0748-321X
• Published Online: 05 June 2017