Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs Using Graph Propagation

Scheinert, D., Zhu, H., Thamsen, L., Geldenhuys, M. K., Will, J., Acker, A. and Kao, O. (2021) Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs Using Graph Propagation. In: 2021 IEEE International Performance, Computing, and Communications Conference (IPCCC), 29-31 Oct 2021, ISBN 9781665443319 (doi: 10.1109/IPCCC51483.2021.9679361)

Abstract

Distributed dataflow systems like Spark and Flink enable the use of clusters for scalable data analytics. While runtime prediction models can be used to initially select appropriate cluster resources given target runtimes, the actual runtime performance of dataflow jobs depends on several factors and varies over time. Yet, in many situations, dynamic scaling can be used to meet formulated runtime targets despite significant performance variance.This paper presents Enel, a novel dynamic scaling approach that uses message propagation on an attributed graph to model dataflow jobs and, thus, allows for deriving effective rescaling decisions. For this, Enel incorporates descriptive properties that capture the respective execution context, considers statistics from individual dataflow tasks, and propagates predictions through the job graph to eventually find an optimized new scale-out. Our evaluation of Enel with four iterative Spark jobs shows that our approach is able to identify effective rescaling actions, reacting for instance to node failures, and can be reused across different execution contexts.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Thamsen, Dr Lauritz
• Authors: Scheinert, D., Zhu, H., Thamsen, L., Geldenhuys, M. K., Will, J., Acker, A., and Kao, O.
• College/School: College of Science and Engineering > School of Computing Science
• Publisher: IEEE
• ISSN: 2374-9628
• ISBN: 9781665443319
• Published Online: 20 January 2022
• Copyright Holders: Copyright © 2021 IEEE
• First Published: First published in 2021 IEEE International Performance, Computing, and Communications Conference (IPCCC)
• Publisher Policy: Reproduced in accordance with the publisher copyright policy

Related URLs

• Pre-print