Exploring Data Splitting Strategies for the Evaluation of Recommendation Models

Meng, Z. , Mccreadie, R. , Macdonald, C. and Ounis, I. (2020) Exploring Data Splitting Strategies for the Evaluation of Recommendation Models. In: 14th ACM Conference on Recommender Systems (RecSys 2020), 22-26 Sep 2020, pp. 681-686. ISBN 9781450375832 (doi: 10.1145/3383313.3418479)

[img] Text
222234.pdf - Accepted Version



Effective methodologies for evaluating recommender systems are critical, so that different systems can be compared in a sound manner. A commonly overlooked aspect of evaluating recommender systems is the selection of the data splitting strategy. In this paper, we both show that there is no standard splitting strategy and that the selection of splitting strategy can have a strong impact on the ranking of recommender systems during evaluation. In particular, we perform experiments comparing three common data splitting strategies, examining their impact over seven state-of-the-art recommendation models on two datasets. Our results demonstrate that the splitting strategy employed is an important confounding variable that can markedly alter the ranking of recommender systems, making much of the currently published literature non-comparable, even when the same datasets and metrics are used.

Item Type:Conference Proceedings
Glasgow Author(s) Enlighten ID:Mccreadie, Dr Richard and Meng, Dr Zaiqiao and Macdonald, Professor Craig and Ounis, Professor Iadh
Authors: Meng, Z., Mccreadie, R., Macdonald, C., and Ounis, I.
College/School:College of Science and Engineering > School of Computing Science
Copyright Holders:Copyright © 2020 The Authors
First Published:First published in 14th ACM Conference on Recommender Systems (RecSys 2020): 681-686
Publisher Policy:Reproduced in accordance with the publisher copyright policy
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record

Project CodeAward NoProject NamePrincipal InvestigatorFunder's NameFunder RefLead Dept
300332BigDataStackIadh OunisEuropean Commission (EC)779747Computing Science