Scalable distributed event detection for Twitter

McCreadie, R. , Macdonald, C. , Ounis, I. , Osborne, M. and Petrovic, S. (2013) Scalable distributed event detection for Twitter. In: 2013 IEEE International Conference on Big Data, Santa Clara, CA, USA, 6-9 Oct 2013, (doi: 10.1109/BigData.2013.6691620)

Abstract

Social media streams, such as Twitter, have shown themselves to be useful sources of real-time information about what is happening in the world. Automatic detection and tracking of events identified in these streams have a variety of real-world applications, e.g. identifying and automatically reporting road accidents for emergency services. However, to be useful, events need to be identified within the stream with a very low latency. This is challenging due to the high volume of posts within these social streams. In this paper, we propose a novel event detection approach that can both effectively detect events within social streams like Twitter and can scale to thousands of posts every second. Through experimentation on a large Twitter dataset, we show that our approach can process the equivalent to the full Twitter Firehose stream, while maintaining event detection accuracy and outperforming an alternative distributed event detection system.

• Item Type: Conference Proceedings
• Keywords: System analysis and design, event detection, distributed processing, large-scale systems, scalability, storm, Big Data
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Mccreadie, Dr Richard and Macdonald, Professor Craig and Ounis, Professor Iadh
• Authors: McCreadie, R., Macdonald, C., Ounis, I., Osborne, M., and Petrovic, S.
• College/School: College of Science and Engineering > School of Computing Science
• Research Group: School of Computing Science - Terrier Team
• Copyright Holders: Copyright © 2013 IEEE
• Publisher Policy: Reproduced in accordance with the copyright policy of the publisher