Phoebe: QoS-Aware Distributed Stream Processing through Anticipating Dynamic Workloads

Geldenhuys, M. K., Scheinert, D., Kao, O. and Thamsen, L. (2022) Phoebe: QoS-Aware Distributed Stream Processing through Anticipating Dynamic Workloads. In: IEEE International Conference on Web Services (ICWS 2022), Barcelona, Spain, 11-15 July 2022, pp. 198-207. ISBN 9781665481434 (doi: 10.1109/ICWS55610.2022.00041)

Abstract

Distributed Stream Processing systems have become an essential part of big data processing platforms. They are characterized by the high-throughput processing of near to real-time event streams with the goal of delivering low-latency results and thus enabling time-sensitive decision making. At the same time, results are expected to be consistent even in the presence of partial failures where exactly-once processing guarantees are required for correctness. Stream processing workloads are oftentimes dynamic in nature which makes static configurations highly inefficient as time goes by. Static resource allocations will almost certainly either negatively impact upon the Quality of Service and/or result in higher operational costs.In this paper we present Phoebe, a proactive approach to system auto-tuning for Distributed Stream Processing jobs executing on dynamic workloads. Our approach makes use of parallel profiling runs, QoS modeling, and runtime optimization to provide a general solution whereby configuration parameters are automatically tuned to ensure a stable service as well as alignment with recovery time Quality of Service targets. Phoebe makes use of Time Series Forecasting to gain an insight into future workload requirements thereby delivering scaling decisions which are accurate, long-lived, and reliable. Our experiments demonstrate that Phoebe is able to deliver a stable service while at the same time reducing resource over-provisioning.

• Item Type: Conference Proceedings
• Additional Information: This work has been supported through grants by the German Ministry for Education and Research (BMBF) as BIFOLD (funding mark 01IS18025A) and WaterGridSense 4.0 (funding mark 02WIK1475D).
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Thamsen, Dr Lauritz
• Authors: Geldenhuys, M. K., Scheinert, D., Kao, O., and Thamsen, L.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9781665481434
• Copyright Holders: Copyright © 2022 IEEE
• Publisher Policy: Reproduced in accordance with the copyright policy of the publisher

Related URLs

• Organisation
• Publisher