Identifying Careless Workers in Crowdsourcing Platforms

Moshfeghi, Y., Huertas-Rosero, A. F. and Jose, J. M. (2016) Identifying Careless Workers in Crowdsourcing Platforms. In: 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, Pisa, Italy, 17-21 Jul 2016, pp. 857-860. ISBN 9781450340694 (doi: 10.1145/2911451.2914756)

Abstract

In this paper we introduce a game scenario for crowdsourcing (CS) using incentives as a bait for careless (gambler) workers, who respond to them in a characteristic way. We hypothesise that careless workers are risk-inclined and can be detected in the game scenario by their use of time, and test this hypothesis in two steps: first, we formulate and prove a theorem stating that a risk-inclined worker will react to competition with shorter Task Completion Time (TCT) than a risk-neutral or risk-averse worker. Second, we check if the game scenario introduces a link between TCT and performance, by performing a crowdsourced evaluation using 35 topics from the TREC-8 collection. Experimental evidence confirms our hypothesis, showing that TCT can be used as a powerful discrimination factor to detect careless workers. This is a valuable result in the quest for quality assurance in CS-based micro tasks such as relevance assessment.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Jose, Professor Joemon and Moshfeghi, Dr Yashar
• Authors: Moshfeghi, Y., Huertas-Rosero, A. F., and Jose, J. M.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9781450340694

Related URLs

• Organisation