Ellis: Dynamically Scaling Distributed Dataflows to Meet Runtime Targets

Thamsen, L., Verbitskiy, I., Beilharz, J., Renner, T., Polze, A. and Kao, O. (2017) Ellis: Dynamically Scaling Distributed Dataflows to Meet Runtime Targets. In: 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), Hong Kong, China, 11-14 Dec 2017, pp. 146-153. ISBN 9781538606926 (doi: 10.1109/CloudCom.2017.37)

Abstract

Distributed dataflow systems like MapReduce, Spark, and Flink help users in analyzing large datasets with a set of cluster resources. Performance modeling and runtime prediction is then used for automatically allocating resources for specific performance goals. However, the actual performance of distributed dataflow jobs can vary significantly due to factors like interference with co-located workloads, varying degrees of data locality, and failures. We address this problem with Ellis, a system that allocates an initial set of resources for a specific runtime target, yet also continuously monitors a job's progress towards the target and if necessary dynamically adjusts the allocation. For this, Ellis models the scale-out behavior of individual stages of distributed dataflow jobs based on previous executions. Our evaluation of Ellis with iterative Spark jobs shows that dynamic adjustments can reduce the number of constraint violations by 30.7-75.0% and the magnitude of constraint violations by 70.6-94.5%.

• Item Type: Conference Proceedings
• Additional Information: Funding: This work has been supported through grants by the German Science Foundation (DFG) as FOR 1306 Stratosphere and by the German Ministry for Education and Research (BMBF) as Berlin Big Data Center BBDC (funding mark 01IS14013A).
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Thamsen, Dr Lauritz
• Authors: Thamsen, L., Verbitskiy, I., Beilharz, J., Renner, T., Polze, A., and Kao, O.
• College/School: College of Science and Engineering > School of Computing Science
• Publisher: IEEE
• ISBN: 9781538606926