Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts

Scheinert, D., Thamsen, L., Zhu, H., Will, J., Acker, A., Wittkopp, T. and Kao, O. (2021) Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts. In: 2021 IEEE International Conference on Cluster Computing (CLUSTER), 07-10 Sep 2021, pp. 261-270. ISBN 9781728196664 (doi: 10.1109/Cluster48925.2021.00052)

Abstract

Distributed dataflow systems enable the use of clusters for scalable data analytics. However, selecting appropriate cluster resources for a processing job is often not straightforward. Performance models trained on historical executions of a concrete job are helpful in such situations, yet they are usually bound to a specific job execution context (e.g. node type, software versions, job parameters) due to the few considered input parameters. Even in case of slight context changes, such supportive models need to be retrained and cannot benefit from historical execution data from related contexts.This paper presents Bellamy, a novel modeling approach that combines scale-outs, dataset sizes, and runtimes with additional descriptive properties of a dataflow job. It is thereby able to capture the context of a job execution. Moreover, Bellamy is realizing a two-step modeling approach. First, a general model is trained on all the available data for a specific scalable analytics algorithm, hereby incorporating data from different contexts. Subsequently, the general model is optimized for the specific situation at hand, based on the available data for the concrete context. We evaluate our approach on two publicly available datasets consisting of execution data from various dataflow jobs carried out in different environments, showing that Bellamy outperforms state-of-the-art methods.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Thamsen, Dr Lauritz
• Authors: Scheinert, D., Thamsen, L., Zhu, H., Will, J., Acker, A., Wittkopp, T., and Kao, O.
• College/School: College of Science and Engineering > School of Computing Science
• Publisher: IEEE
• ISSN: 2168-9253
• ISBN: 9781728196664
• Published Online: 13 October 2021
• Copyright Holders: Copyright © 2021 IEEE
• First Published: First published in 2021 IEEE International Conference on Cluster Computing (CLUSTER): 261-270
• Publisher Policy: Reproduced in accordance with the publisher copyright policy

Related URLs

• Pre-print