Is Multi-Modal Data Key for Crisis Content Categorization on Social Media?

Long, Z. and McCreadie, R. (2022) Is Multi-Modal Data Key for Crisis Content Categorization on Social Media? In: 19th International Conference on Information Systems for Crisis Response and Management (ISCRAM 2022), Tarbes, France, 22-25 May 2022, (In Press)

Abstract

The user-base of social media platforms, like Twitter, has grown dramatically around the world over the last decade. As people post everything they experience on social media, large volumes of valuable multimedia content are being recorded online, which can be analysed to help for a range of tasks. Here we specifically focus on crisis response. The majority of prior works in this space focus on using machine learning to categorize single-modality content (e.g. text of the posts, or images shared), with few works jointly utilizing multiple modalities. Hence, in this paper, we examine to what extent integrating multiple modalities is important for crisis content categorization. In particular, we design a pipeline for multi-modal learning that fuses textual and visual inputs, leverages both, and then classifies that content based on the specified task. Through evaluation using the CrisisMMD dataset, we demonstrate that effective automatic labelling for this task is possible, with an average of 88.31\% F1 performance across two significant tasks (relevance and humanitarian category classification). while also analysing cases that unimodal models and multi-modal models success and fail.

• Item Type: Conference Proceedings
• Status: In Press
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: LONG, ZIJUN and Mccreadie, Dr Richard
• Authors: Long, Z., and McCreadie, R.
• Subjects: Q Science > Q Science (General)
• College/School: College of Science and Engineering > School of Computing Science
• Research Centre: College of Science and Engineering > School of Computing Science > IDA Section
• Research Group: Information Retrieval

Related URLs

• Organisation
• Publisher