Algorithmic Bias

Wilkie, C. and Azzopardi, L. (2017) Algorithmic Bias. In: ACM Conference On Information and Knowledge Management (CIKM '17), Singapore, 06-10 Nov 2017, 2375- 2378. ISBN 9781450349185 (doi:10.1145/3132847.3133135)

[img]
Preview
Text
151477.pdf - Accepted Version

341kB

Abstract

Algorithmic bias presents a difficult challenge within Information Retrieval. Long has it been known that certain algorithms favour particular documents due to attributes of these documents that are not directly related to relevance. The evaluation of bias has recently been made possible through the use of retrievability, a quantifiable measure of bias. While evaluating bias is relatively novel, the evaluation of performance has been common since the dawn of the Cranfield approach and TREC. To evaluate performance, a pool of documents to be judged by human assessors is created from the collection. This pooling approach has faced accusations of bias due to the fact that the state of the art algorithms were used to create it, thus the inclusion of biases associated with these algorithms may be included in the pool. The introduction of retrievability has provided a mechanism to evaluate the bias of these pools. This work evaluates the varying degrees of bias present in the groups of relevant and non-relevant documents for topics. The differentiating power of a system is also evaluated by examining the documents from the pool that are retrieved for each topic. The analysis finds that the systems that perform better, tend to have a higher chance of retrieving a relevant document rather than a non-relevant document for a topic prior to retrieval, indicating that retrieval systems which perform better at TREC are already predisposed to agree with the judgements regardless of the query posed.

Item Type:Conference Proceedings
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Azzopardi, Dr Leif and Wilkie, Mr Colin
Authors: Wilkie, C., and Azzopardi, L.
College/School:College of Science and Engineering > School of Computing Science
Journal Name:Proceedings of the 2017 ACM on Conference on Information and Knowledge Management - CIKM '17
Publisher:ACM Press
ISBN:9781450349185
Copyright Holders:Copyright © 2017 ACM
Publisher Policy:Reproduced in accordance with the copyright policy of the publisher

University Staff: Request a correction | Enlighten Editors: Update this record

Project CodeAward NoProject NamePrincipal InvestigatorFunder's NameFunder RefLead Dept
642131Doctoral Training Grant 2013 - 2017Mary Beth KneafseyEngineering and Physical Sciences Research Council (EPSRC)EP/L50497X/1RSI - RESEARCH STRATEGY & INNOVATION