The Common Workflow Scheduler Interface: Status Quo and Future Plans

Lehmann, F., Bader, J., Thamsen, L. and Leser, U. (2023) The Common Workflow Scheduler Interface: Status Quo and Future Plans. Workshop on Workflows in Support of Large-Scale Science (WORKS23),The International Conference on High Performance Computing, Network, Storage, and Analysis, New York, USA, 12-17 Nov 2023. ISBN 9798400707858 (doi: 10.1145/3624062.3626283)

Abstract

Nowadays, many scientific workflows from different domains, such as Remote Sensing, Astronomy, and Bioinformatics, are executed on large computing infrastructures managed by resource managers. Scientific workflow management systems (SWMS) support the workflow execution and communicate with the infrastructures’ resource managers. However, the communication between SWMS and resource managers is complicated by a) inconsistent interfaces between SMWS and resource managers and b) the lack of support for workflow dependencies and workflow-specific properties. To tackle these issues, we developed the Common Workflow Scheduler Interface (CWSI), a simple yet powerful interface to exchange workflow-related information between a SWMS and a resource manager, making the resource manager workflow-aware. The first prototype implementations show that the CWSI can reduce the makespan already with simple but workflow-aware strategies up to 25%. In this paper, we show how existing workflow resource management research can be integrated into the CWSI.

• Item Type: Conference or Workshop Item
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Thamsen, Dr Lauritz
• Authors: Lehmann, F., Bader, J., Thamsen, L., and Leser, U.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9798400707858
• Copyright Holders: Copyright: © 2023 The Authors
• First Published: First published in SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis: 2100-2102
• Publisher Policy: Reproduced in accordance with the publisher copyright policy

Related URLs

• Other