Retrievability: an evaluation measure for higher order information access tasks

Azzopardi, L. and Vinay, V. (2008) Retrievability: an evaluation measure for higher order information access tasks. In: 17th ACM Conference on Information and Knowledge Management, Napa Valley, California, USA, 26-30 Oct 2008, pp. 561-570. ISBN 9781595939913 (doi: 10.1145/1458082.1458157)

Abstract

Evaluation in Information Retrieval (IR) has long focused on effectiveness and efficiency. However, new and emerging access tasks now demand alternative evaluation measures which go beyond this traditional view. A retrieval system provides a means of gaining access to documents, therefore intuitively, our view of the collection is shaped by the retrieval system. In this paper, we outline some emerging information access related scenarios that require knowledge about how the retrieval system affects the users' ability to access information. This provides the motivation for the proposed evaluation measures and methodology where the focus is on capturing the behavior of the system, in terms of how retrievable it makes individual documents within the collection. To demonstrate the utility of the proposed methods, we perform an extensive analysis on two TREC collections showing how the measures can be applied to evaluate different information access questions. For higher order information access tasks that are inherently dependent on retrievability, our novel evaluation methodology emphasizes that effectiveness is an insufficient characterization of a retrieval system. This paper provides the foundations for the evaluation of higher order access related tasks.

• Item Type: Conference Proceedings
• Keywords: accessibililty, retrievability
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Azzopardi, Dr Leif
• Authors: Azzopardi, L., and Vinay, V.
• Subjects: Z Bibliography. Library Science. Information Resources > Z665 Library Science. Information Science; Q Science > QA Mathematics > QA75 Electronic computers. Computer science
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9781595939913