Phoebe: QoS-Aware Distributed Stream Processing through Anticipating Dynamic Workloads

Geldenhuys, M. K., Scheinert, D., Kao, O. and Thamsen, L. (2022) Phoebe: QoS-Aware Distributed Stream Processing through Anticipating Dynamic Workloads. In: IEEE International Conference on Web Services (ICWS 2022), Barcelona, Spain, 11-15 July 2022, pp. 198-207. ISBN 9781665481434 (doi: 10.1109/ICWS55610.2022.00041)

[img] Text
272111.pdf - Accepted Version



Distributed Stream Processing systems have become an essential part of big data processing platforms. They are characterized by the high-throughput processing of near to real-time event streams with the goal of delivering low-latency results and thus enabling time-sensitive decision making. At the same time, results are expected to be consistent even in the presence of partial failures where exactly-once processing guarantees are required for correctness. Stream processing workloads are oftentimes dynamic in nature which makes static configurations highly inefficient as time goes by. Static resource allocations will almost certainly either negatively impact upon the Quality of Service and/or result in higher operational costs.In this paper we present Phoebe, a proactive approach to system auto-tuning for Distributed Stream Processing jobs executing on dynamic workloads. Our approach makes use of parallel profiling runs, QoS modeling, and runtime optimization to provide a general solution whereby configuration parameters are automatically tuned to ensure a stable service as well as alignment with recovery time Quality of Service targets. Phoebe makes use of Time Series Forecasting to gain an insight into future workload requirements thereby delivering scaling decisions which are accurate, long-lived, and reliable. Our experiments demonstrate that Phoebe is able to deliver a stable service while at the same time reducing resource over-provisioning.

Item Type:Conference Proceedings
Additional Information:This work has been supported through grants by the German Ministry for Education and Research (BMBF) as BIFOLD (funding mark 01IS18025A) and WaterGridSense 4.0 (funding mark 02WIK1475D).
Glasgow Author(s) Enlighten ID:Thamsen, Dr Lauritz
Authors: Geldenhuys, M. K., Scheinert, D., Kao, O., and Thamsen, L.
College/School:College of Science and Engineering > School of Computing Science
Copyright Holders:Copyright © 2022 IEEE
Publisher Policy:Reproduced in accordance with the copyright policy of the publisher
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record