Incident Streams 2021 off the Deep End: Deeper Annotations and Evaluations in Twitter

Buntain, C., McCreadie, R. and Soboroff, I. (2022) Incident Streams 2021 off the Deep End: Deeper Annotations and Evaluations in Twitter. In: 19th International Conference on Information Systems for Crisis Response and Management (ISCRAM 2022), Tarbes, France, 22-25 May 2022, pp. 584-604. ISBN 9788284270999 (doi: http://idl.iscram.org/files/codybuntain/2022/2441_CodyBuntain_etal2022.pdf)

[img] Text
279527.pdf - Accepted Version
Restricted to Repository staff only

4MB

Abstract

This paper summarizes the final year of the four-year Incident Streams track (TREC-IS), which has produced a large dataset comprising 136,263 annotated tweets, spanning 98 crisis events. Goals of this final year were twofold: 1) to add new categories for assessing messages, with a focus on characterizing the audience, author, and images associated with these messages, and 2) to significantly enlarge the TREC-IS dataset with new events, with an emphasis of deeper pools for sampling. Beyond these two goals, TREC-IS has nearly doubled the number of annotated messages per event for the 26 crises introduced in 2021 and has released a new parallel dataset of 312,546 images associated with crisis content -- with 7,297 tweets having annotations about their embedded images. Our analyses of this new crisis data yields new insights about the context of a tweet; e.g., messages intended for a local audience and those that contain images of weather forecasts and infographics have higher than average assessments of priority but are relatively rare. Tweets containing images, however, have significantly higher perceived priorities than tweets without images. Moving to deeper pools, while tending to lower classification performance, also does not generally impact performance rankings or alter distributions of information-types. We end this paper with a discussion of these datasets, analyses, their implications, and how they contribute both new data and insights to the broader crisis informatics community.

Item Type:Conference Proceedings
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Mccreadie, Dr Richard
Authors: Buntain, C., McCreadie, R., and Soboroff, I.
Subjects:Q Science > Q Science (General)
College/School:College of Science and Engineering > School of Computing Science
Research Centre:College of Science and Engineering > School of Computing Science > IDA Section
Research Group:Information Retrieval
ISSN:2411-3387
ISBN:9788284270999
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record