Reshi: Recommending Resources for Scientific Workflow Tasks on Heterogeneous Infrastructures

Bader, J., Lehmann, F., Groth, A., Thamsen, L., Scheinert, D., Will, J., Leser, U. and Kao, O. (2022) Reshi: Recommending Resources for Scientific Workflow Tasks on Heterogeneous Infrastructures. In: 41st IEEE International Performance, Computing, and Communications Conference (IPCCC 2022), Austin, TX, USA, 11-13 November 2022, pp. 269-274. ISBN 9781665480185 (doi: 10.1109/IPCCC55026.2022.9894299)

Abstract

Scientific workflows typically comprise a multitude of different processing steps which often are executed in parallel on different partitions of the input data. These executions, in turn, must be scheduled on the compute nodes of the computational infrastructure at hand. This assignment is complicated by the facts that (a) tasks typically have highly heterogeneous resource requirements and (b) in many infrastructures, compute nodes offer highly heterogeneous resources. In consequence, predictions of the runtime of a given task on a given node, as required by many scheduling algorithms, are often rather imprecise, which can lead to sub-optimal scheduling decisions.We propose Reshi, a method for recommending task-node assignments during workflow execution that can cope with heterogeneous tasks and heterogeneous nodes. Reshi approaches the problem as a regression task, where task-node pairs are modeled as feature vectors over the results of dedicated micro benchmarks and past task executions. Based on these features, Reshi trains a regression tree model to rank and recommend nodes for each ready-to-run task, which can be used as input to a scheduler. For our evaluation, we benchmarked 27 AWS machine types using three representative workflows. We compare Reshi’s recommendations with three state-of-the-art schedulers. Our evaluation shows that Reshi outperforms HEFT by a mean makespan reduction of 7.18% and 18.01% assuming a mean task runtime prediction error of 15%.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Thamsen, Dr Lauritz
• Authors: Bader, J., Lehmann, F., Groth, A., Thamsen, L., Scheinert, D., Will, J., Leser, U., and Kao, O.
• College/School: College of Science and Engineering > School of Computing Science
• ISSN: 2374-9628
• ISBN: 9781665480185
• Copyright Holders: Copyright © 2022 IEEE
• Publisher Policy: Reproduced in accordance with the copyright policy of the publisher

Related URLs

• Organisation
• Publisher