Predicting Dynamic Memory Requirements for Scientific Workflow Tasks

Bader, J., Diedrich, N., Thamsen, L. and Kao, O. (2024) Predicting Dynamic Memory Requirements for Scientific Workflow Tasks. In: 2023 IEEE International Conference on Big Data, Sorrento, Italy, 15-18 Dec 2023, ISBN 9798350324457 (doi: 10.1109/BigData59044.2023.10386837)

Abstract

With the increasing amount of data available to scientists in disciplines as diverse as bioinformatics, physics, and remote sensing, scientific workflow systems are becoming increasingly important for composing and executing scalable data analysis pipelines. When writing such workflows, users need to specify the resources to be reserved for tasks so that sufficient resources are allocated on the target cluster infrastructure. Crucially, underestimating a task’s memory requirements can result in task failures. Therefore, users often resort to overprovisioning, resulting in significant resource wastage and decreased throughput. In this paper, we propose a novel online method that uses monitoring time series data to predict task memory usage in order to reduce the memory wastage of scientific workflow tasks. Our method predicts a task’s runtime, divides it into k equally-sized segments, and learns the peak memory value for each segment depending on the total file input size. We evaluate the prototype implementation of our method using workflows from the publicly available nf-core repository, showing an average memory wastage reduction of 29.48% compared to the best state-of-the-art approach.

• Item Type: Conference Proceedings
• Additional Information: Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) as FONDA (Project 414984028, SFB 1404).
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Thamsen, Dr Lauritz
• Authors: Bader, J., Diedrich, N., Thamsen, L., and Kao, O.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9798350324457
• Copyright Holders: Copyright © 2023, IEEE
• First Published: First published in 2023 IEEE International Conference on Big Data (BigData)
• Publisher Policy: Reproduced in accordance with the publisher copyright policy

Related URLs

• Other