A Comparison of Nuggets and Clusters for Evaluating Timeline Summaries

Baruah, G., Mccreadie, R. and Lin, J. (2017) A Comparison of Nuggets and Clusters for Evaluating Timeline Summaries. In: ACM Conference On Information and Knowledge Management (CIKM '17), Singapore, 06-10 Nov 2017, pp. 67-76. ISBN 9781450349185 (doi:10.1145/3132847.3133000)

[img]
Preview
Text
148594.pdf - Accepted Version

5MB

Abstract

There is growing interest in systems that generate timeline summaries by filtering high-volume streams of documents to retain only those that are relevant to a particular event or topic. Continued advances in algorithms and techniques for this task depend on standardized and reproducible evaluation methodologies for comparing systems. However, timeline summary evaluation is still in its infancy, with competing methodologies currently being explored in international evaluation forums such as TREC. One area of active exploration is how to explicitly represent the units of information that should appear in a 'good' summary. Currently, there are two main approaches, one based on identifying nuggets in an external 'ground truth', and the other based on clustering system outputs. In this paper, by building test collections that have both nugget and cluster annotations, we are able to compare these two approaches. Specifically, we address questions related to evaluation effort, differences in the final evaluation products, and correlations between scores and rankings generated by both approaches. We summarize advantages and disadvantages of nuggets and clusters to offer recommendations for future system evaluations

Item Type:Conference Proceedings
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Mccreadie, Dr Richard
Authors: Baruah, G., Mccreadie, R., and Lin, J.
Subjects:Q Science > QA Mathematics > QA75 Electronic computers. Computer science
College/School:College of Science and Engineering > School of Computing Science
Research Group:Information retrieval
ISBN:9781450349185
Copyright Holders:Copyright © 2017 The Authors
First Published:First published in ACM Conference On Information and Knowledge Management (CIKM '17): 67-76
Publisher Policy:Reproduced in accordance with the publisher copyright policy

University Staff: Request a correction | Enlighten Editors: Update this record

Project CodeAward NoProject NamePrincipal InvestigatorFunder's NameFunder RefLead Dept
624701SUPERIadh OunisEuropean Commission (EC)606853COM - COMPUTING SCIENCE