On the Potential of Execution Traces for Batch Processing Workload Optimization in Public Clouds

Scheinert, D., Alamgiralem, A., Bader, J., Will, J., Wittkopp, T. and Thamsen, L. (2021) On the Potential of Execution Traces for Batch Processing Workload Optimization in Public Clouds. In: 2021 IEEE International Conference on Big Data (Big Data), 15-18 Dec 2021, pp. 3113-3118. ISBN 9781665439022 (doi: 10.1109/BigData52589.2021.9671275)

[img] Text
268167.pdf - Accepted Version

431kB

Abstract

With the growing amount of data, data processing workloads and the management of their resource usage becomes increasingly important. Since managing a dedicated infrastructure is in many situations infeasible or uneconomical, users progressively execute their respective workloads in the cloud. As the configuration of workloads and resources is often challenging, various methods have been proposed that either quickly profile towards a good configuration or determine one based on data from previous runs. Still, performance data to train such methods is often lacking and must be costly collected.In this paper, we propose a collaborative approach for sharing anonymized workload execution traces among users, mining them for general patterns, and exploiting clusters of historical workloads for future optimizations. We evaluate our prototype implementation for mining workload execution graphs on a publicly available trace dataset and demonstrate the predictive value of workload clusters determined through traces only.

Item Type:Conference Proceedings
Additional Information:The 5th Workshop on Benchmarking, Performance Tuning and Optimization for Big Data Applications (BPOD).
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Thamsen, Dr Lauritz
Authors: Scheinert, D., Alamgiralem, A., Bader, J., Will, J., Wittkopp, T., and Thamsen, L.
College/School:College of Science and Engineering > School of Computing Science
Publisher:IEEE
ISBN:9781665439022
Published Online:13 January 2022
Copyright Holders:Copyright © 2021 The Authors
Publisher Policy:Reproduced in accordance with the publisher copyright policy
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record