How Sensitivity Classification Effectiveness Impacts Reviewers in Technology-Assisted Sensitivity Review

Mcdonald, G. , Macdonald, C. and Ounis, I. (2019) How Sensitivity Classification Effectiveness Impacts Reviewers in Technology-Assisted Sensitivity Review. In: ACM SIGIR Conference on Human Information Interaction & Retrieval (CHIIR), Glasgow, UK, 10-14 Mar 2019, pp. 337-341. ISBN 9781450360258 (doi: 10.1145/3295750.3298962)

[img]
Preview
Text
174446.pdf - Accepted Version

827kB

Abstract

All government documents that are released to the public must first be manually reviewed to identify and protect any sensitive information, e.g. confidential information. However, the unassisted manual sensitivity review of born-digital documents is not practical due to, for example, the volume of documents that are created. Previous work has shown that sensitivity classification can be effective for predicting if a document contains sensitive information. However, since all of the released documents must be manually reviewed, it is important to know if sensitivity classification can assist sensitivity reviewers in making their sensitivity judgements. Hence, in this paper, we conduct a digital sensitivity review user study, to investigate if the accuracy of sensitivity classification effects the number of documents that a reviewer correctly judges to be sensitive or not (reviewer accuracy) and the time that it takes to sensitivity review a document (reviewing speed). Our results show that providing reviewers with sensitivity classification predictions, from a classifier that achieves 0.7 Balanced Accuracy, results in a 38% increase in mean reviewer accuracy and an increase of 72% in mean reviewing speeds, compared to when reviewers are not provided with predictions. Overall, our findings demonstrate that sensitivity classification is a viable technology for assisting with the sensitivity review of born-digital government documents.

Item Type:Conference Proceedings
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Macdonald, Professor Craig and McDonald, Dr Graham and Ounis, Professor Iadh
Authors: Mcdonald, G., Macdonald, C., and Ounis, I.
College/School:College of Science and Engineering > School of Computing Science
ISBN:9781450360258
Copyright Holders:Copyright © 2019 The Authors
Publisher Policy:Reproduced in accordance with the copyright policy of the publisher
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record