Effectively Testing System Configurations of Critical IoT Analytics Pipelines

Geldenhuys, M. K., Thamsen, L., Gontarska, K. K., Lorenz, F. and Kao, O. (2020) Effectively Testing System Configurations of Critical IoT Analytics Pipelines. In: 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA, 09-12 Dec 2019, pp. 4157-4162. ISBN 9781728108582 (doi: 10.1109/BigData47090.2019.9005504)

Abstract

The emergence of the Internet of Things has seen the introduction of numerous connected devices used for the monitoring and control of even Critical Infrastructures. Distributed stream processing has become key to analyzing data generated by these connected devices and improving our ability to make decisions. However, optimizing these systems towards specific Quality of Service targets is a difficult and time-consuming task, due to the large-scale distributed systems involved, the existence of so many configuration parameters, and the inability to easily determine the impact of tuning these parameters. In this paper we present an approach for the effective testing of system configurations for critical IoT analytics pipelines. We demonstrate our approach with a prototype that we called Timon which is integrated with Kubernetes. This tool allows pipelines to be easily replicated in parallel and evaluated to determine the optimal configuration for specific applications. We demonstrate the usefulness of our approach by investigating different configurations of an exemplary geographically-based traffic monitoring application implemented in Apache Flink.

• Item Type: Conference Proceedings
• Additional Information: Funding: This work has been supported through grants by the German Ministry for Education and Research (BMBF) as Berlin Big Data Center BBDC2 (funding mark 01IS18025A).
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Thamsen, Dr Lauritz
• Authors: Geldenhuys, M. K., Thamsen, L., Gontarska, K. K., Lorenz, F., and Kao, O.
• College/School: College of Science and Engineering > School of Computing Science
• Publisher: IEEE
• ISBN: 9781728108582

Related URLs

• Pre-print