Benchmarking News Recommendations in a Living Lab

Hopfgartner, F. , Kille, B., Lommatzsch, A., Plumbaum, T., Brodt, T. and Heintz, T. (2014) Benchmarking News Recommendations in a Living Lab. In: CLEF'14: Proceedings of the 5th International Conference of the CLEF Initiative, Sheffield, UK, 15-18 Sept 2014, pp. 250-267. ISBN 9783319113821

Abstract

Most user-centric studies of information access systems in literature suffer from unrealistic settings or limited numbers of users who participate in the study. In order to address this issue, the idea of a living lab has been promoted. Living labs allow us to evaluate research hypotheses using a large number of users who satisfy their information need in a real context. In this paper, we introduce a living lab on news recommendation in real time. The living lab has first been organized as News Recommendation Challenge at ACM RecSys’13 and then as campaign-style evaluation lab NEWSREEL at CLEF’14. Within this lab, researchers were asked to provide news article recommendations to millions of users in real time. Different from user studies which have been performed in a laboratory, these users are following their own agenda. Consequently, laboratory bias on their behavior can be neglected. We outline the living lab scenario and the experimental setup of the two benchmarking events. We argue that the living lab can serve as reference point for the implementation of living labs for the evaluation of information access systems.

• Item Type: Conference Proceedings
• Additional Information: Published in volume 8685 of the series Lecture Notes in Computer Science pp 250-267.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Hopfgartner, Dr Frank
• Authors: Hopfgartner, F., Kille, B., Lommatzsch, A., Plumbaum, T., Brodt, T., and Heintz, T.
• Subjects: Z Bibliography. Library Science. Information Resources > Z665 Library Science. Information Science
• College/School: College of Arts & Humanities > School of Humanities > Information Studies
• Publisher: Springer Verlag
• ISBN: 9783319113821
• Publisher Policy: Reproduced in accordance with the publisher policy