Detecting and resolving inconsistencies between domain experts’ different perspectives on (classification) tasks

Sleeman, D., Moss, L. , Aiken, A., Hughes, M., Kinsella, J. and Sim, M. (2012) Detecting and resolving inconsistencies between domain experts’ different perspectives on (classification) tasks. Artificial Intelligence in Medicine, 55(2), pp. 71-86. (doi: 10.1016/j.artmed.2012.03.001)

Abstract

Objectives: The work reported here focuses on developing novel techniques which enable an expert to detect inconsistencies in 2 (or more) perspectives that the expert might have on the same (classification) task. The high level task which the experts (physicians) had set themselves was to classify, on a 5-point severity scale (A–E), the hourly reports produced by an intensive care unit's patient management system. <p/>Method: The INSIGHT system has been developed to support domain experts exploring, and removing inconsistencies in their conceptualization of a task. We report here a study of intensive care physicians reconciling 2 perspectives on their patients. The 2 perspectives provided to INSIGHT were an annotated set of patient records where the expert had selected the appropriate category to describe that snapshot of the patient, and a set of rules which are able to classify the various time points on the same 5-point scale. Inconsistencies between these 2 perspectives are displayed as a confusion matrix; moreover INSIGHT then allows the expert to revise both the annotated datasets (correcting data errors, or changing the assigned categories) and the actual rule-set. <p/>Results: Each of the 3 experts achieved a very high degree of consensus (∼97%) between his refined knowledge sources (i.e., annotated hourly patient records and the rule-set). We then had the experts produce a common rule-set and then refine their several sets of annotations against it; this again resulted in inter-expert agreements of ∼97%. The resulting rule-set can then be used in applications with considerable confidence. <p/>Conclusion: This study has shown that under some circumstances, it is possible for domain experts to achieve a high degree of correlation between 2 perspectives of the same task. The experts agreed that the immediate feedback provided by INSIGHT was a significant contribution to this successful outcome.

• Item Type: Articles
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Sim, Malcolm and Kinsella, Professor John and Moss, Dr Laura and Sleeman, Prof Derek and Hughes, Dr Martin
• Authors: Sleeman, D., Moss, L., Aiken, A., Hughes, M., Kinsella, J., and Sim, M.
• College/School: College of Medical Veterinary and Life Sciences > School of Medicine, Dentistry & Nursing; College of Medical Veterinary and Life Sciences > School of Medicine, Dentistry & Nursing > Clinical Specialities
• Journal Name: Artificial Intelligence in Medicine
• ISSN: 0933-3657
• ISSN (Online): 1873-2860
• Published Online: 04 April 2012