Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs Using Graph Propagation

Scheinert, D., Zhu, H., Thamsen, L., Geldenhuys, M. K., Will, J., Acker, A. and Kao, O. (2021) Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs Using Graph Propagation. In: 2021 IEEE International Performance, Computing, and Communications Conference (IPCCC), 29-31 Oct 2021, ISBN 9781665443319 (doi: 10.1109/IPCCC51483.2021.9679361)

[img] Text
268166.pdf - Accepted Version

536kB

Abstract

Distributed dataflow systems like Spark and Flink enable the use of clusters for scalable data analytics. While runtime prediction models can be used to initially select appropriate cluster resources given target runtimes, the actual runtime performance of dataflow jobs depends on several factors and varies over time. Yet, in many situations, dynamic scaling can be used to meet formulated runtime targets despite significant performance variance.This paper presents Enel, a novel dynamic scaling approach that uses message propagation on an attributed graph to model dataflow jobs and, thus, allows for deriving effective rescaling decisions. For this, Enel incorporates descriptive properties that capture the respective execution context, considers statistics from individual dataflow tasks, and propagates predictions through the job graph to eventually find an optimized new scale-out. Our evaluation of Enel with four iterative Spark jobs shows that our approach is able to identify effective rescaling actions, reacting for instance to node failures, and can be reused across different execution contexts.

Item Type:Conference Proceedings
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Thamsen, Dr Lauritz
Authors: Scheinert, D., Zhu, H., Thamsen, L., Geldenhuys, M. K., Will, J., Acker, A., and Kao, O.
College/School:College of Science and Engineering > School of Computing Science
Publisher:IEEE
ISSN:2374-9628
ISBN:9781665443319
Published Online:20 January 2022
Copyright Holders:Copyright © 2021 IEEE
First Published:First published in 2021 IEEE International Performance, Computing, and Communications Conference (IPCCC)
Publisher Policy:Reproduced in accordance with the publisher copyright policy
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record