Idomaar: A Framework for Multi-dimensional Benchmarking of Recommender Algorithms

Scriminaci, M., Lommatzsch, A., Kille, B., Hopfgartner, F. , Larson, M., Malagoli, D. and Sereny, A. (2016) Idomaar: A Framework for Multi-dimensional Benchmarking of Recommender Algorithms. 10th ACM Conference on Recommender Systems, Baltimore, MA, USA, 15-19 Sept 2016.

Abstract

In real-world scenarios, recommenders face non-functional requirements of technical nature and must handle dynamic data in the form of sequential streams. Evaluation of recommender systems must take these issues into account in order to be maximally informative. In this paper, we present Idomaar—a framework that enables the efficient multi-dimensional benchmarking of recommender algorithms. Idomaar goes beyond current academic research practices by creating a realistic evaluation environment and computing both effectiveness and technical metrics for stream-based as well as setbased evaluation. A scenario focussing on “research to prototyping to productization” cycle at a company illustrates Idomaar’s potential. We show that Idomaar simplifies testing with varying configurations and supports flexible integration of different data.

• Item Type: Conference or Workshop Item
• Additional Information: The research leading to these results was performed in the CrowdRec project, which has received funding from the EU 7th Framework Programme FP7/2007-2013 under grant agreement No. 610594.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Hopfgartner, Dr Frank
• Authors: Scriminaci, M., Lommatzsch, A., Kille, B., Hopfgartner, F., Larson, M., Malagoli, D., and Sereny, A.
• College/School: College of Arts & Humanities > School of Humanities > Information Studies
• Copyright Holders: Copyright © 2016 ACM
• Publisher Policy: Reproduced in accordance with the copyright policy of the publisher

Related URLs

• Organisation