Relying on topic subsets for system ranking estimation

Hauff, C., Hiemstra, D., Jong, F. and Azzopardi, L. (2009) Relying on topic subsets for system ranking estimation. In: 18th ACM Conference on Information and Knowledge Management, Hong Kong, 2-6 Nov 2009, pp. 1859-1862. ISBN 9781605585123 (doi: 10.1145/1645953.1646249)

Full text not currently available from Enlighten.

Publisher's URL: http://portal.acm.org/citation.cfm?id=1645953.1646249

Abstract

Ranking a number of retrieval systems according to their retrieval effectiveness without relying on costly relevance judgments was first explored by Soboroff et al [6]. Over the years, a number of alternative approaches have been proposed. We perform a comprehensive analysis of system ranking estimation approaches on a wide variety of TREC test collections and topics sets. Our analysis reveals that the performance of such approaches is highly dependent upon the topic or topic subset, used for estimation. We hypothesize that the performance of system ranking estimation approaches can be improved by selecting the "right" subset of topics and show that using topic subsets improves the performance by 32% on average, with a maximum improvement of up to 70% in some cases.

Item Type:Conference Proceedings
Keywords:system ranking estimation, evaluation
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Azzopardi, Dr Leif
Authors: Hauff, C., Hiemstra, D., Jong, F., and Azzopardi, L.
Subjects:Z Bibliography. Library Science. Information Resources > Z665 Library Science. Information Science
Q Science > QA Mathematics > QA75 Electronic computers. Computer science
College/School:College of Science and Engineering > School of Computing Science
ISBN:9781605585123

University Staff: Request a correction | Enlighten Editors: Update this record