How Sensitive is Recommendation Systems' Offline Evaluation to Popularity?

Jadidinejad, A. , Macdonald, C. and Ounis, I. (2019) How Sensitive is Recommendation Systems' Offline Evaluation to Popularity? In: REVEAL 2019 Workshop at RecSys, Copenhagen, Denmark, 20 Sep 2019,

193202.pdf - Accepted Version



Datasets used for the offline evaluation of recommender systems are collected through user interactions with an already deployed recommender system. However, such datasets can be subject to different types of biases including a system’s popularity bias. In this paper, we focus on assessing the influence of popularity on the offline evaluation of recommendation systems. Our insights from a deeper analysis based on popularity-stratified sampling reveal that the current offline evaluation of recommendation systems are sensitive to popular items, raising questions about conclusions driven from the offline comparison of recommendation models.

Item Type:Conference Proceedings
Glasgow Author(s) Enlighten ID:Jadidinejad, Dr Amir and Macdonald, Professor Craig and Ounis, Professor Iadh
Authors: Jadidinejad, A., Macdonald, C., and Ounis, I.
College/School:College of Science and Engineering > School of Computing Science
Copyright Holders:Copyright © 2019 The Authors
Publisher Policy:Reproduced with the permission of the publisher
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record

Project CodeAward NoProject NamePrincipal InvestigatorFunder's NameFunder RefLead Dept
3009820Exploiting Closed-Loop Aspects in Computationally and Data Intensive AnalyticsRoderick Murray-SmithEngineering and Physical Sciences Research Council (EPSRC)EP/R018634/1Computing Science