CrisisFACTS: Building and Evaluating Crisis Timelines

Mccreadie, R. and Buntain, C. (2023) CrisisFACTS: Building and Evaluating Crisis Timelines. In: 20th International Conference on Information Systems for Crisis Response and Management (ISCRAM 2023), Omaha, NE, USA, 28-31 May 2023, pp. 320-339. ISBN 9798218217495 (doi: 10.59297/JVQZ9405)

[img] Text
295806.pdf - Accepted Version

3MB

Abstract

Between 2018 and 2021, the Incident Streams track (TREC-IS) developed standard approaches for classifying information types and criticality of tweets during crises. While successful in producing substantial collections of labeled data, TREC-IS as a data challenge had several limitations: It only evaluated information at type-level rather than what was reported; it only used Twitter data; and it lacked measures of redundancy in system output. This paper introduces Crisis Facts and Cross-Stream Temporal Summarization (CrisisFACTS), a new data challenge piloted in 2022 and developed to address these limitations. The CrisisFACTS framework recasts TREC-IS into an event-summarization task using multiple disaster-relevant data streams and a new fact-based evaluation scheme, allowing the community to assess state-of-the-art methods for summarizing disaster events. Results from CrisisFACTS in 2022 include a new test-collection comprising human-generated disaster summaries along with multi-platform datasets of social media, crisis reports and news coverage for major crisis events.

Item Type:Conference Proceedings
Keywords:Emergency management, crisis informatics, news, Twitter, Facebook, Reddit, Wikipedia, summarization.
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Mccreadie, Dr Richard
Authors: Mccreadie, R., and Buntain, C.
Subjects:Q Science > QA Mathematics > QA75 Electronic computers. Computer science
College/School:College of Science and Engineering > School of Computing Science
Research Centre:College of Science and Engineering > School of Computing Science > IDA Section > GPU Cluster
Research Group:Information Retrieval
ISSN:2411-3387
ISBN:9798218217495
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record