A Study of Realtime Summarization Metrics

Ekstrand-Abueg, M., McCreadie, R. , Pavlu, V. and Diaz, F. (2016) A Study of Realtime Summarization Metrics. In: 25th ACM International Conference on Information and Knowledge Management (CIKM 2016), Indianapolis, IN, USA, 24-28 Oct 2016, pp. 2125-2130. ISBN 9781450340731 (doi:10.1145/2983323.2983653)

148597.pdf - Accepted Version



Unexpected news events, such as natural disasters or other human tragedies, create a large volume of dynamic text data from official news media as well as less formal social media. Automatic real-time text summarization has become an important tool for quickly transforming this overabundance of text into clear, useful information for end-users including affected individuals, crisis responders, and interested third parties. Despite the importance of real-time summarization systems, their evaluation is not well understood as classic methods for text summarization are inappropriate for real-time and streaming conditions. The TREC 2013-2015 Temporal Summarization (TREC-TS) track was one of the first evaluation campaigns to tackle the challenges of real-time summarization evaluation, introducing new metrics, ground-truth generation methodology and dataset. In this paper, we present a study of TREC-TS track evaluation methodology, with the aim of documenting its design, analyzing its effectiveness, as well as identifying improvements and best practices for the evaluation of temporal summarization systems.

Item Type:Conference Proceedings
Glasgow Author(s) Enlighten ID:Mccreadie, Dr Richard
Authors: Ekstrand-Abueg, M., McCreadie, R., Pavlu, V., and Diaz, F.
Subjects:Q Science > QA Mathematics > QA75 Electronic computers. Computer science
College/School:College of Science and Engineering > School of Computing Science
Research Group:Information Retrieval
Copyright Holders:Copyright © 2016 The Authors
First Published:First published in CIKM '16 Proceedings of the 25th ACM International on Conference on Information and Knowledge Management: 2125-2130
Publisher Policy:Reproduced in accordance with the publisher copyright policy

University Staff: Request a correction | Enlighten Editors: Update this record