Towards Collaborative Optimization of Cluster Configurations for Distributed Dataflow Jobs

Will, J., Bader, J. and Thamsen, L. (2020) Towards Collaborative Optimization of Cluster Configurations for Distributed Dataflow Jobs. In: 2020 IEEE International Conference on Big Data (Big Data), 10-13 Dec 2020, pp. 2851-2856. ISBN 9781728162515 (doi: 10.1109/BigData50022.2020.9377994)

[img] Text
268152.pdf - Accepted Version

389kB

Abstract

Analyzing large datasets with distributed dataflow systems requires the use of clusters. Public cloud providers offer a large variety and quantity of resources that can be used for such clusters. However, picking the appropriate resources in both type and number can often be challenging, as the selected configuration needs to match a distributed dataflow job's resource demands and access patterns. A good cluster configuration avoids hardware bottlenecks and maximizes resource utilization, avoiding costly overprovisioning.We propose a collaborative approach for finding optimal cluster configurations based on sharing and learning from historical runtime data of distributed dataflow jobs. Collaboratively shared data can be utilized to predict runtimes of future job executions through the use of specialized regression models. However, training prediction models on historical runtime data that were produced by different users and in diverse contexts requires the models to take these contexts into account.

Item Type:Conference Proceedings
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Thamsen, Dr Lauritz
Authors: Will, J., Bader, J., and Thamsen, L.
College/School:College of Science and Engineering > School of Computing Science
Publisher:IEEE
ISBN:9781728162515
Published Online:19 March 2021
Copyright Holders:Copyright © 2020 IEEE
First Published:First published in 2020 IEEE International Conference on Big Data (Big Data): 2851-2856
Publisher Policy:Reproduced in accordance with the publisher copyright policy
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record