An in-depth study on diversity evaluation: the importance of intrinsic diversity

Yu, H.-T., Jatowt, A., Blanco, R., Joho, H. and Jose, J. M. (2017) An in-depth study on diversity evaluation: the importance of intrinsic diversity. Information Processing and Management, 53(4), pp. 799-813. (doi: 10.1016/j.ipm.2017.03.001)

[img]
Preview
Text
137948.pdf - Accepted Version
Available under License Creative Commons Attribution Non-commercial No Derivatives.

2MB

Abstract

Diversified document ranking has been recognized as an effective strategy to tackle ambiguous and/or underspecified queries. In this paper, we conduct an in-depth study on diversity evaluation that provides insights for assessing the performance of a diversified retrieval system. By casting the widely used diversity metrics (e.g., ERR-IA, α-nDCG and D#-nDCG) into a unified framework based on marginal utility, we analyze how these metrics capture extrinsic diversity and intrinsic diversity. Our analyses show that the prior metrics (ERR-IA, α-nDCG and D#-nDCG) are not able to precisely measure intrinsic diversity if we merely feed a set of subtopics into them in a traditional manner (i.e., without fine-grained relevance knowledge per subtopic). As the redundancy of relevant documents with respect to each specific information need (i.e., subtopic) can not be then detected and solved, the overall diversity evaluation may not be reliable. Furthermore, a series of experiments are conducted on a gold standard collection (English and Chinese) and a set of submitted runs, where the intent-square metrics that extend the diversity metrics through incorporating hierarchical subtopics are used as references. The experimental results show that the intent-square metrics disagree with the diversity metrics (ERR-IA and α-nDCG) being used in a traditional way on top-ranked runs, and that the average precision correlation scores between intent-square metrics and the prior diversity metrics (ERR-IA and α-nDCG) are fairly low. These results justify our analyses, and uncover the previously-unknown importance of intrinsic diversity to the overall diversity evaluation.

Item Type:Articles
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Jose, Professor Joemon and Joho, Mr Hideo and Yu, Dr Haitao
Authors: Yu, H.-T., Jatowt, A., Blanco, R., Joho, H., and Jose, J. M.
College/School:College of Science and Engineering > School of Computing Science
Journal Name:Information Processing and Management
Publisher:Elsevier
ISSN:0306-4573
ISSN (Online):1873-5371
Published Online:10 March 2017
Copyright Holders:Copyright © 2017 Elsevier Ltd.
First Published:First published in Information Processing and Management 53(4):799-813
Publisher Policy:Reproduced in accordance with the copyright policy of the publisher

University Staff: Request a correction | Enlighten Editors: Update this record