A comprehensive analysis of parameter settings for novelty-biased cumulative gain

Leelanupab, T., Zuccon, G. and Jose, J. M. (2012) A comprehensive analysis of parameter settings for novelty-biased cumulative gain. In: CIKM 21st ACM International Conference on Information and Knowledge Management, Maui, HI, 29 Oct - 2 Nov 2012, pp. 1950-1954.

Abstract

In the TREC Web Diversity track, novelty-biased cumulative gain (α-NDCG) is one of the official measures to assess retrieval performance of IR systems. The measure is characterised by a parameter, α, the effect of which has not been thoroughly investigated. We find that common settings of α, i.e. α=0.5, may prevent the measure from behaving as desired when evaluating result diversification. This is because it excessively penalises systems that cover many intents while it rewards those that redundantly cover only few intents. This issue is crucial since it highly influences systems at top ranks. We revisit our previously proposed threshold, suggesting α be set on a query-basis. The intuitiveness of the measure is then studied by examining actual rankings from TREC 09-10 Web track submissions. By varying α according to our query-based threshold, the discriminative power of α-NDCG is not harmed and in fact, our approach improves α-NDCG's robustness. Experimental results show that the threshold for α can turn the measure to be more intuitive than using its common settings.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Jose, Professor Joemon and Leelanupab, Mr Teerapong
• Authors: Leelanupab, T., Zuccon, G., and Jose, J. M.
• College/School: College of Science and Engineering > School of Computing Science

Related URLs

• Organisation