Get Your Memory Right: The Crispy Resource Allocation Assistant for Large-Scale Data Processing

Will, J., Thamsen, L., Bader, J., Scheinert, D. and Kao, O. (2022) Get Your Memory Right: The Crispy Resource Allocation Assistant for Large-Scale Data Processing. In: 10th IEEE International Conference on Cloud Engineering (IC2E), California, USA, 26-30 Sept 2022, ISBN 9781665491150 (doi: 10.1109/IC2E55432.2022.00014)

[img] Text
274181.pdf - Accepted Version

432kB

Abstract

Distributed dataflow systems like Apache Spark and Apache Hadoop enable data-parallel processing of large datasets on clusters. Yet, selecting appropriate computational resources for dataflow jobs — that neither lead to bottlenecks nor to low resource utilization — is often challenging, even for expert users such as data engineers. Further, existing automated approaches to resource selection rely on the assumption that a job is recurring to learn from previous runs or to warrant the cost of full test runs to learn from. However, this assumption often does not hold since many jobs are too unique. Therefore, we present Crispy, a method for optimizing data processing cluster configurations based on job profiling runs with small samples of the dataset on just a single machine. Crispy attempts to extrapolate the memory usage for the full dataset to then choose a cluster configuration with enough total memory. In our evaluation on a dataset with 1031 Spark and Hadoop jobs, we see a reduction of job execution costs by 56% compared to the baseline, while on average spending less than ten minutes on profiling runs per job on a consumer-grade laptop.

Item Type:Conference Proceedings
Additional Information:This work has been supported through grants by the German Ministry for Education and Research (BMBF) as BIFOLD (grant 01IS18025A) and the German Research Foundation (DFG) as FONDA (DFG Collaborative Research Center 1404).
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Thamsen, Dr Lauritz
Authors: Will, J., Thamsen, L., Bader, J., Scheinert, D., and Kao, O.
College/School:College of Science and Engineering > School of Computing Science
ISBN:9781665491150
Copyright Holders:Copyright © 2022 IEEE
First Published:First published in 2022 IEEE International Conference on Cloud Engineering (IC2E)
Publisher Policy:Reproduced in accordance with the publisher copyright policy
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record