Selecting Efficient Cluster Resources for Data Analytics: When and How to Allocate for In-Memory Processing?

Will, J., Thamsen, L., Scheinert, D. and Kao, O. (2023) Selecting Efficient Cluster Resources for Data Analytics: When and How to Allocate for In-Memory Processing? In: 35th International Conference on Scientific and Statistical Database Management (SSDBM2023), Los Angeles, California, USA, 10-12 July 2023, ISBN 9798400707469 (doi: 10.1145/3603719.3603733)

Abstract

Distributed dataflow systems such as Apache Spark or Apache Flink enable parallel, in-memory data processing on large clusters of commodity hardware. Consequently, the appropriate amount of memory to allocate to the cluster is a crucial consideration. In this paper, we analyze the challenge of efficient resource allocation for distributed data processing, focusing on memory. We emphasize that in-memory processing with in-memory data processing frameworks can undermine resource efficiency. Based on the findings of our trace data analysis, we compile requirements towards an automated solution for efficient cluster resource allocation.

• Item Type: Conference Proceedings
• Additional Information: This work has been supported through a grant by the German Research Foundation (DFG) as “C5” (grant 506529034).
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Thamsen, Dr Lauritz
• Authors: Will, J., Thamsen, L., Scheinert, D., and Kao, O.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9798400707469
• Copyright Holders: Copyright © 2023 held by the owner/author(s)
• First Published: First published in SSDBM '23: Proceedings of the 35th International Conference on Scientific and Statistical Database Management
• Publisher Policy: Reproduced in accordance with the publisher copyright policy

Related URLs

• Organisation