Get Your Memory Right: The Crispy Resource Allocation Assistant for Large-Scale Data Processing

Will, J., Thamsen, L., Bader, J., Scheinert, D. and Kao, O. (2022) Get Your Memory Right: The Crispy Resource Allocation Assistant for Large-Scale Data Processing. In: 10th IEEE International Conference on Cloud Engineering (IC2E), California, USA, 26-30 Sept 2022, ISBN 9781665491150 (doi: 10.1109/IC2E55432.2022.00014)

Abstract

Distributed dataflow systems like Apache Spark and Apache Hadoop enable data-parallel processing of large datasets on clusters. Yet, selecting appropriate computational resources for dataflow jobs — that neither lead to bottlenecks nor to low resource utilization — is often challenging, even for expert users such as data engineers. Further, existing automated approaches to resource selection rely on the assumption that a job is recurring to learn from previous runs or to warrant the cost of full test runs to learn from. However, this assumption often does not hold since many jobs are too unique. Therefore, we present Crispy, a method for optimizing data processing cluster configurations based on job profiling runs with small samples of the dataset on just a single machine. Crispy attempts to extrapolate the memory usage for the full dataset to then choose a cluster configuration with enough total memory. In our evaluation on a dataset with 1031 Spark and Hadoop jobs, we see a reduction of job execution costs by 56% compared to the baseline, while on average spending less than ten minutes on profiling runs per job on a consumer-grade laptop.

• Item Type: Conference Proceedings
• Additional Information: This work has been supported through grants by the German Ministry for Education and Research (BMBF) as BIFOLD (grant 01IS18025A) and the German Research Foundation (DFG) as FONDA (DFG Collaborative Research Center 1404).
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Thamsen, Dr Lauritz
• Authors: Will, J., Thamsen, L., Bader, J., Scheinert, D., and Kao, O.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9781665491150
• Copyright Holders: Copyright © 2022 IEEE
• First Published: First published in 2022 IEEE International Conference on Cloud Engineering (IC2E)
• Publisher Policy: Reproduced in accordance with the publisher copyright policy

Related URLs

• Publisher