On the Additivity and Weak Baselines for Search Result Diversification Research

Akcay, M., Altingovde, I. S., Macdonald, C. and Ounis, I. (2017) On the Additivity and Weak Baselines for Search Result Diversification Research. In: ICTIR 2017: The 3rd ACM International Conference on the Theory of Information Retrieval, Amsterdam, The Netherlands, 1-4 Oct 2017, pp. 109-116. ISBN 9781450344906 (doi: 10.1145/3121050.3121059)

[img]
Preview
Text
145956.pdf - Accepted Version

862kB

Abstract

A recent study on the topic of additivity addresses the task of search result diversification and concludes that while weaker baselines are almost always significantly improved by the evaluated diversification methods, for stronger baselines, just the opposite happens, i.e., no significant improvement can be observed. Due to the importance of the issue in shaping future research directions and evaluation strategies in search results diversification, in this work, we first aim to reproduce the findings reported in the previous study, and then investigate its possible limitations. Our extensive experiments first reveal that under the same experimental setting with that previous study, we can reach similar results. Next, we hypothesize that for stronger baselines, tuning the parameters of some methods (i.e., the trade-off parameter between the relevance and diversity of the results in this particular scenario) should be done in a more fine-grained manner. With trade-off parameters that are specifically determined for each baseline run, we show that the percentage of significant improvements even over the strong baselines can be doubled. As a further issue, we discuss the possible impact of using the same strong baseline retrieval function for the diversity computations of the methods. Our takeaway message is that in the case of a strong baseline, it is more crucial to tune the parameters of the diversification methods to be evaluated; but once this is done, additivity is achievable.

Item Type:Conference Proceedings
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Macdonald, Professor Craig and Ounis, Professor Iadh
Authors: Akcay, M., Altingovde, I. S., Macdonald, C., and Ounis, I.
College/School:College of Science and Engineering > School of Computing Science
ISBN:9781450344906
Copyright Holders:Copyright © 2017 ACM
First Published:First published in ICTIR '17 Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval: 109-116
Publisher Policy:Reproduced in accordance with the publisher copyright policy
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record