University of Glasgow (qirdcsuog) at TREC Crowdsourcing 2011: TurkRank-Network-based worker ranking in crowdsourcing

Whiting, S., Perez, J. R., Zuccon, G., Leelanupab, T. and Jose, J. M. (2011) University of Glasgow (qirdcsuog) at TREC Crowdsourcing 2011: TurkRank-Network-based worker ranking in crowdsourcing. In: TREC (Text REtrieval Conference) 2011, Gaithersburg, MD, USA, 15-18 Nov 2011,

Abstract

For TREC Crowdsourcing 2011 (Stage 2) we propose a networkbased approach for assigning an indicative measure of worker trustworthiness in crowdsourced labelling tasks. Workers, the gold standard and worker/gold standard agreements are modelled as a network. For the purpose of worker trustworthiness assignment, a variant of the PageRank algorithm, named TurkRank, is used to adaptively combine evidence that suggests worker trustworthiness, i.e., agreement with other trustworthy co-workers and agreement with the gold standard. A single parameter controls the importance of co-worker agreement versus gold standard agreement. The TurkRank score calculated for each worker is incorporated with a worker-weighted mean label aggregation.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Jose, Professor Joemon and Whiting, Mr Stewart and Leelanupab, Mr Teerapong and Zuccon, Mr Guido
• Authors: Whiting, S., Perez, J. R., Zuccon, G., Leelanupab, T., and Jose, J. M.
• College/School: College of Science and Engineering > School of Computing Science