Scaling out Big Data Missing Value Imputations: Pythia vs. Godzilla

Anagnostopoulos, C. and Triantafillou, P. (2014) Scaling out Big Data Missing Value Imputations: Pythia vs. Godzilla. In: 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '14), New York, N.Y., U.S.A, 24-27 Aug 2014, pp. 651-660. ISBN 9781450329569 (doi:10.1145/2623330.2623615)

Full text not currently available from Enlighten.

Abstract

Solving the missing-value (MV) problem with small estimation errors in big data environments is a notoriously resource-demanding task. As datasets and their user community continuously grow, the problem can only be exacerbated. Assume that it is possible to have a single machine (`Godzilla'), which can store the massive dataset and support an ever-growing community submitting MV imputation requests. Is it possible to replace Godzilla by employing a large number of cohort machines so that imputations can be performed much faster, engaging cohorts in parallel, each of which accesses much smaller partitions of the original dataset? If so, it would be preferable for obvious performance reasons to access only a subset of all cohorts per imputation. In this case, can we decide swiftly which is the desired subset of cohorts to engage per imputation? But efficiency and scalability is just one key concern! Is it possible to do the above while ensuring comparable or even better than Godzilla's imputation estimation errors? In this paper we derive answers to these fundamentals questions and develop principled methods and a framework which offer large performance speed-ups and better, or comparable, errors to that of Godzilla, independently of which missing-value imputation algorithm is used. Our contributions involve Pythia, a framework and algorithms for providing the answers to the above questions and for engaging the appropriate subset of cohorts per MV imputation request. Pythia functionality rests on two pillars: (i) dataset (partition) signatures, one per cohort, and (ii) similarity notions and algorithms, which can identify the appropriate subset of cohorts to engage. Comprehensive experimentation with real and synthetic datasets showcase our efficiency, scalability, and accuracy claims.

Item Type:Conference Proceedings
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Anagnostopoulos, Dr Christos and Triantafillou, Professor Peter
Authors: Anagnostopoulos, C., and Triantafillou, P.
Subjects:Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Q Science > QA Mathematics > QA76 Computer software
College/School:College of Science and Engineering > School of Computing Science
ISBN:9781450329569

University Staff: Request a correction | Enlighten Editors: Update this record