How Workflow Engines Should Talk to Resource Managers: A Proposal for a Common Workflow Scheduling Interface

Lehmann, F., Bader, J., Tschirpke, F., Thamsen, L. and Leser, U. (2023) How Workflow Engines Should Talk to Resource Managers: A Proposal for a Common Workflow Scheduling Interface. In: 23rd IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGrid 2023), Bangalore, India, 1-4 May 2023, pp. 166-179. ISBN 9798350301199 (doi: 10.1109/CCGrid57682.2023.00025)

Abstract

Scientific workflow management systems (SWMSs) and resource managers together ensure that tasks are scheduled on provisioned resources so that all dependencies are obeyed, and some optimization goal, such as makespan minimization, is achieved. In practice, however, there is no clear separation of scheduling responsibilities between an SWMS and a resource manager because there exists no agreed-upon separation of concerns between their different components. This has two consequences. First, the lack of a standardized API to exchange scheduling information between SWMSs and resource managers hinders portability. It incurs costly adaptations when a component should be replaced by a different one (e.g., an SWMS with another SWMS on the same resource manager). Second, due to overlapping functionalities, current installations often actually have two schedulers, both making partial scheduling decisions under incomplete information, leading to suboptimal workflow scheduling. In this paper, we propose a simple REST interface between SWMSs and resource managers, which allows any SWMS to pass dynamic workflow information to a resource manager, enabling maximally informed scheduling decisions. We provide an implementation of this API as an example, using Nextflow as an SWMS and Kubernetes as a resource manager. Our experiments with nine real-world workflows show that this strategy reduces makespan by up to 25.1% and 10.8% on average compared to the standard Nextflow/Kubernetes configuration. Furthermore, a more widespread implementation of this API would enable leaner code bases, a simpler exchange of components of workflow systems, and a unified place to implement new scheduling algorithms.

• Item Type: Conference Proceedings
• Additional Information: This work was funded by the German Research Foundation (DFG), CRC 1404: ”FONDA: Foundations of Workflows for Large-Scale Scientific Data Analysis.”
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Thamsen, Dr Lauritz
• Authors: Lehmann, F., Bader, J., Tschirpke, F., Thamsen, L., and Leser, U.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9798350301199
• Copyright Holders: Copyright © 2023, IEEE
• First Published: First published in 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)
• Publisher Policy: Reproduced in accordance with the publisher copyright policy

Related URLs

• Organisation