Collaborative cluster configuration for distributed data-parallel processing: a research overview

Thamsen, L., Scheinert, D., Will, J., Bader, J. and Kao, O. (2022) Collaborative cluster configuration for distributed data-parallel processing: a research overview. Datenbank-Spektrum, 22(2), pp. 143-151. (doi: 10.1007/s13222-022-00416-z)

[img] Text
272120.pdf - Published Version
Available under License Creative Commons Attribution.

989kB

Abstract

Many organizations routinely analyze large datasets using systems for distributed data-parallel processing and clusters of commodity resources. Yet, users need to configure adequate resources for their data processing jobs. This requires significant insights into expected job runtimes and scaling behavior, resource characteristics, input data distributions, and other factors. Unable to estimate performance accurately, users frequently overprovision resources for their jobs, leading to low resource utilization and high costs. In this paper, we present major building blocks towards a collaborative approach for optimization of data processing cluster configurations based on runtime data and performance models. We believe that runtime data can be shared and used for performance models across different execution contexts, significantly reducing the reliance on the recurrence of individual processing jobs or, else, dedicated job profiling. For this, we describe how the similarity of processing jobs and cluster infrastructures can be employed to combine suitable data points from local and global job executions into accurate performance models. Furthermore, we outline approaches to performance prediction via more context-aware and reusable models. Finally, we lay out how metrics from previous executions can be combined with runtime monitoring to effectively re-configure models and clusters dynamically.

Item Type:Articles
Additional Information:This work has been supported through grants by the German Ministry for Education and Research (BMBF) as BIFOLD (grant 01IS18025A) and the German Research Foundation (DFG) as FONDA (DFG Collaborative Research Center 1404). Open Access funding enabled and organized by Projekt DEAL.
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Thamsen, Dr Lauritz
Authors: Thamsen, L., Scheinert, D., Will, J., Bader, J., and Kao, O.
College/School:College of Science and Engineering > School of Computing Science
Journal Name:Datenbank-Spektrum
Publisher:Springer
ISSN:1618-2162
ISSN (Online):1610-1995
Published Online:31 May 2022
Copyright Holders:Copyright © 2022 The Authors
First Published:First published in Datenbank-Spektrum 22(2): 143-151
Publisher Policy:Reproduced under a Creative Commons License

University Staff: Request a correction | Enlighten Editors: Update this record