How Sensitive is Recommendation Systems' Offline Evaluation to Popularity?

Jadidinejad, A. , Macdonald, C. and Ounis, I. (2019) How Sensitive is Recommendation Systems' Offline Evaluation to Popularity? In: REVEAL 2019 Workshop at RecSys, Copenhagen, Denmark, 20 Sep 2019,

Abstract

Datasets used for the offline evaluation of recommender systems are collected through user interactions with an already deployed recommender system. However, such datasets can be subject to different types of biases including a system’s popularity bias. In this paper, we focus on assessing the influence of popularity on the offline evaluation of recommendation systems. Our insights from a deeper analysis based on popularity-stratified sampling reveal that the current offline evaluation of recommendation systems are sensitive to popular items, raising questions about conclusions driven from the offline comparison of recommendation models.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Jadidinejad, Dr Amir and Macdonald, Professor Craig and Ounis, Professor Iadh
• Authors: Jadidinejad, A., Macdonald, C., and Ounis, I.
• College/School: College of Science and Engineering > School of Computing Science
• Copyright Holders: Copyright © 2019 The Authors
• Publisher Policy: Reproduced with the permission of the publisher

Related URLs

• Organisation

Project Code	Award No	Project Name	Principal Investigator	Funder's Name	Funder Ref	Lead Dept
300982	0	Exploiting Closed-Loop Aspects in Computationally and Data Intensive Analytics	Roderick Murray-Smith	Engineering and Physical Sciences Research Council (EPSRC)	EP/R018634/1	Computing Science