Resolving Differences of Opinion between Medical Experts: A Case Study with the IS-DELPHI System

Sleeman, D., Kostadinov, K., Moss, L. and Sim, M. (2020) Resolving Differences of Opinion between Medical Experts: A Case Study with the IS-DELPHI System. In: 13th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2020), Valletta, Malta, 24-26 Feb 2020, pp. 66-76. ISBN 9789897583988 (doi: 10.5220/0008922000660076)

Abstract

Knowledge intensive clinical systems, as well as machine learning algorithms, have become more widely used over the last decade or so. These systems often need access to sizable labelled datasets which could be more useful if their instances are accurately labelled / annotated. A variety of approaches, including statistical ones, have been used to label instances. In this paper, we discuss the use of domain experts, in this case clinicians, to perform this task. Here we recognize that even highly rated domain experts can have differences of opinion on certain instances; we discuss a system inspired by the Delphi approaches which helps experts resolve their differences of opinion on classification tasks. The focus of this paper is the IS-DELPHI tool which we have implemented to address the labelling issue; we report its use in a medical domain in a study involving 12 Intensive Care Unit clinicians. The several pairs of experts initially disagreed on the classification of 11 instances but as a result of using IS-DELPHI all those disagreements were resolved. From participant feedback (questionnaires), we have concluded that the medical experts understood the task and were comfortable with the functionality provided by IS-DELPHI. We plan to further enhance the system’s capabilities and usability, and then use IS-DELPHI, which is a domain independent tool, in a number of further medical domains.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Sim, Malcolm and Moss, Dr Laura
• Authors: Sleeman, D., Kostadinov, K., Moss, L., and Sim, M.
• College/School: College of Medical Veterinary and Life Sciences > School of Medicine, Dentistry & Nursing
• ISSN: 2184-4305
• ISBN: 9789897583988