Exploring Data Splitting Strategies for the Evaluation of Recommendation Models

Meng, Z. , Mccreadie, R. , Macdonald, C. and Ounis, I. (2020) Exploring Data Splitting Strategies for the Evaluation of Recommendation Models. In: 14th ACM Conference on Recommender Systems (RecSys 2020), 22-26 Sep 2020, pp. 681-686. ISBN 9781450375832 (doi: 10.1145/3383313.3418479)

Abstract

Effective methodologies for evaluating recommender systems are critical, so that different systems can be compared in a sound manner. A commonly overlooked aspect of evaluating recommender systems is the selection of the data splitting strategy. In this paper, we both show that there is no standard splitting strategy and that the selection of splitting strategy can have a strong impact on the ranking of recommender systems during evaluation. In particular, we perform experiments comparing three common data splitting strategies, examining their impact over seven state-of-the-art recommendation models on two datasets. Our results demonstrate that the splitting strategy employed is an important confounding variable that can markedly alter the ranking of recommender systems, making much of the currently published literature non-comparable, even when the same datasets and metrics are used.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Mccreadie, Dr Richard and Meng, Dr Zaiqiao and Macdonald, Professor Craig and Ounis, Professor Iadh
• Authors: Meng, Z., Mccreadie, R., Macdonald, C., and Ounis, I.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9781450375832
• Copyright Holders: Copyright © 2020 The Authors
• First Published: First published in 14th ACM Conference on Recommender Systems (RecSys 2020): 681-686
• Publisher Policy: Reproduced in accordance with the publisher copyright policy

Related URLs

• Organisation

Project Code	Award No	Project Name	Principal Investigator	Funder's Name	Funder Ref	Lead Dept
300332		BigDataStack	Iadh Ounis	European Commission (EC)	779747	Computing Science