Chiron: Optimizing Fault Tolerance in QoS-aware Distributed Stream Processing Jobs

Geldenhuys, M., Thamsen, L. and Kao, O. (2020) Chiron: Optimizing Fault Tolerance in QoS-aware Distributed Stream Processing Jobs. In: 2020 IEEE International Conference on Big Data (Big Data), 10-13 Dec 2020, pp. 434-440. ISBN 9781728162515 (doi: 10.1109/BigData50022.2020.9378474)

[img] Text
268151.pdf - Accepted Version



Fault tolerance is a property which needs deeper consideration when dealing with streaming jobs requiring high levels of availability and low-latency processing even in case of failures where Quality-of-Service constraints must be adhered to. Typically, systems achieve fault tolerance and the ability to recover automatically from partial failures by implementing Checkpoint and Rollback Recovery. However, this is an expensive operation which impacts negatively on the overall performance of the system and manually optimizing fault tolerance for specific jobs is a difficult and time consuming task.In this paper we introduce Chiron, an approach for automatically optimizing the frequency with which checkpoints are performed in streaming jobs. For any chosen job, parallel profiling runs are performed, each containing a variant of the configurations, with the resulting metrics used to model the impact of checkpoint-based fault tolerance on performance and availability. Understanding these relationships is key to minimizing performance objectives and meeting strict Quality-of-Service constraints. We implemented Chiron prototypically together with Apache Flink and demonstrate its usefulness experimentally.

Item Type:Conference Proceedings
Glasgow Author(s) Enlighten ID:Thamsen, Dr Lauritz
Authors: Geldenhuys, M., Thamsen, L., and Kao, O.
College/School:College of Science and Engineering > School of Computing Science
Published Online:19 March 2021
Copyright Holders:Copyright © 2020 IEEE
First Published:First published in 2020 IEEE International Conference on Big Data (Big Data): 434-440
Publisher Policy:Reproduced in accordance with the publisher copyright policy
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record