Identifying Careless Workers in Crowdsourcing Platforms

Moshfeghi, Y., Huertas-Rosero, A. F. and Jose, J. M. (2016) Identifying Careless Workers in Crowdsourcing Platforms. In: 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, Pisa, Italy, 17-21 Jul 2016, pp. 857-860. ISBN 9781450340694 (doi: 10.1145/2911451.2914756)

Full text not currently available from Enlighten.


In this paper we introduce a game scenario for crowdsourcing (CS) using incentives as a bait for careless (gambler) workers, who respond to them in a characteristic way. We hypothesise that careless workers are risk-inclined and can be detected in the game scenario by their use of time, and test this hypothesis in two steps: first, we formulate and prove a theorem stating that a risk-inclined worker will react to competition with shorter Task Completion Time (TCT) than a risk-neutral or risk-averse worker. Second, we check if the game scenario introduces a link between TCT and performance, by performing a crowdsourced evaluation using 35 topics from the TREC-8 collection. Experimental evidence confirms our hypothesis, showing that TCT can be used as a powerful discrimination factor to detect careless workers. This is a valuable result in the quest for quality assurance in CS-based micro tasks such as relevance assessment.

Item Type:Conference Proceedings
Glasgow Author(s) Enlighten ID:Jose, Professor Joemon and Moshfeghi, Dr Yashar
Authors: Moshfeghi, Y., Huertas-Rosero, A. F., and Jose, J. M.
College/School:College of Science and Engineering > School of Computing Science
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record