Magpie: Automatically Tuning Static Parameters for Distributed File Systems using Deep Reinforcement Learning

Zhu, H., Scheinert, D., Thamsen, L., Gontarska, K. and Kao, O. (2022) Magpie: Automatically Tuning Static Parameters for Distributed File Systems using Deep Reinforcement Learning. In: 10th IEEE International Conference on Cloud Engineering (IC2E), California, USA, 26-30 Sept 2022, pp. 150-159. ISBN 9781665491150 (doi: 10.1109/IC2E55432.2022.00023)

Abstract

Distributed file systems are widely used nowadays, yet using their default configurations is often not optimal. At the same time, tuning configuration parameters is typically challenging and time-consuming. It demands expertise and tuning operations can also be expensive. This is especially the case for static parameters, where changes take effect only after a restart of the system or workloads. We propose a novel approach, Magpie, which utilizes deep re-inforcement learning to tune static parameters by strategically ex-ploring and exploiting configuration parameter spaces. To boost the tuning of the static parameters, our method employs both server and client metrics of distributed file systems to understand the relationship between static parameters and performance. Our empirical evaluation results show that Magpie can noticeably improve the performance of the distributed file system Lustre, where our approach on average achieves 91.8 % throughput gains against default configuration after tuning towards single performance indicator optimization, while it reaches 39.7% more throughput gains against the baseline.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Thamsen, Dr Lauritz
• Authors: Zhu, H., Scheinert, D., Thamsen, L., Gontarska, K., and Kao, O.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9781665491150
• Copyright Holders: Copyright © 2022 IEEE
• First Published: First published in 2022 IEEE International Conference on Cloud Engineering (IC2E)
• Publisher Policy: Reproduced in accordance with the publisher copyright policy

Related URLs

• Organisation
• Publisher