Garbage collection auto-tuning for Java MapReduce on Multi-Cores

Singer, J. , Kovoor, G., Brown, G. and Lujan, M. (2011) Garbage collection auto-tuning for Java MapReduce on Multi-Cores. In: ISMM '11 International Symposium on Memory Management, San Jose, CA, USA, 4-5 June, 2011, pp. 109-118. (doi: 10.1145/1993478.1993495)

Abstract

MapReduce has been widely accepted as a simple programming pattern that can form the basis for efficient, large-scale, distributed data processing. The success of the MapReduce pattern has led to a variety of implementations for different computational scenarios. In this paper we present MRJ, a MapReduce Java framework for multi-core architectures. We evaluate its scalability on a four-core, hyperthreaded Intel Core i7 processor, using a set of standard MapReduce benchmarks. We investigate the significant impact that Java runtime garbage collection has on the performance and scalability of MRJ. We propose the use of memory management auto-tuning techniques based on machine learning. With our auto-tuning approach, we are able to achieve MRJ performance within 10% of optimal on 75% of our benchmark tests.

• Item Type: Conference Proceedings
• Additional Information: © ACM, 2011. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ISMM '11 Proceedings of the International Symposium on Memory Management http://doi.acm.org/10.1145/2076022.1993495"
• Keywords: mapreduce, garbage collection, machine learning, Java
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Singer, Dr Jeremy
• Authors: Singer, J., Kovoor, G., Brown, G., and Lujan, M.
• Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
• College/School: College of Science and Engineering > School of Computing Science
• Research Group: ENDS
• Copyright Holders: Copyright © 2011 ACM
• First Published: First published in ISMM '11 Proceedings of the International Symposium on Memory Management
• Publisher Policy: Reproduced in accordance with the copyright policy of the publisher