Scheduling Stream Processing Tasks on Geo-Distributed Heterogeneous Resources

Janßen, G., Verbitskiy, I., Renner, T. and Thamsen, L. (2019) Scheduling Stream Processing Tasks on Geo-Distributed Heterogeneous Resources. In: 2018 IEEE International Conference on Big Data (Big Data), Seattle, WA, USA, 10-13 Dec 2018, pp. 5159-5164. ISBN 9781538650356 (doi: 10.1109/BigData.2018.8622651)

Abstract

Low-latency processing of data streams from distributed sensors is becoming increasingly important for a growing number of IoT applications. In these environments sensor data collected at the edge of the network is typically transmitted in a number of hops: from devices to intermediate resources to clusters of cloud resources. Scheduling processing tasks of dataflow jobs on all the resources of these environments can significantly reduce application latencies and network congestion. However, for this schedulers need to take the heterogeneity of processing resources and network topologies into account. This paper examines multiple methods for scheduling distributed dataflow tasks on geo-distributed, heterogeneous resources. For this, we developed an optimization function that incorporates the latencies, bandwidths, and computational resources of heterogeneous topologies. We evaluated the different placement methods in a virtual geo-distributed and heterogeneous environment with an IoT application. Our results show that metaheuristic methods that take service quality metrics into account can find significantly better placements than methods that only take topologies into account, with latencies reduced by almost 50%.

• Item Type: Conference Proceedings
• Additional Information: Funding: This work has been supported through grants by the German Ministry for Education and Research (BMBF) as Berlin Big Data Center BBDC (funding mark 01IS14013A and 01IS18025A).
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Thamsen, Dr Lauritz
• Authors: Janßen, G., Verbitskiy, I., Renner, T., and Thamsen, L.
• College/School: College of Science and Engineering > School of Computing Science
• Publisher: IEEE
• ISBN: 9781538650356