Garbage collection auto-tuning for Java MapReduce on Multi-Cores

Singer, J. , Kovoor, G., Brown, G. and Lujan, M. (2011) Garbage collection auto-tuning for Java MapReduce on Multi-Cores. In: ISMM '11 International Symposium on Memory Management, San Jose, CA, USA, 4-5 June, 2011, pp. 109-118. (doi: 10.1145/1993478.1993495)




MapReduce has been widely accepted as a simple programming pattern that can form the basis for efficient, large-scale, distributed data processing. The success of the MapReduce pattern has led to a variety of implementations for different computational scenarios. In this paper we present MRJ, a MapReduce Java framework for multi-core architectures. We evaluate its scalability on a four-core, hyperthreaded Intel Core i7 processor, using a set of standard MapReduce benchmarks. We investigate the significant impact that Java runtime garbage collection has on the performance and scalability of MRJ. We propose the use of memory management auto-tuning techniques based on machine learning. With our auto-tuning approach, we are able to achieve MRJ performance within 10% of optimal on 75% of our benchmark tests.

Item Type:Conference Proceedings
Additional Information:© ACM, 2011. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ISMM '11 Proceedings of the International Symposium on Memory Management"
Keywords:mapreduce, garbage collection, machine learning, Java
Glasgow Author(s) Enlighten ID:Singer, Dr Jeremy
Authors: Singer, J., Kovoor, G., Brown, G., and Lujan, M.
Subjects:Q Science > QA Mathematics > QA75 Electronic computers. Computer science
College/School:College of Science and Engineering > School of Computing Science
Research Group:ENDS
Copyright Holders:Copyright © 2011 ACM
First Published:First published in ISMM '11 Proceedings of the International Symposium on Memory Management
Publisher Policy:Reproduced in accordance with the copyright policy of the publisher

University Staff: Request a correction | Enlighten Editors: Update this record