A game theory approach for effective crowdsource based relevance assessment

Moshfeghi, Y., Rosero, A. F. H. and Jose, J. (2016) A game theory approach for effective crowdsource based relevance assessment. ACM Transactions on Intelligent Systems and Technology, 7(4), 55. (doi: 10.1145/2873063)

Full text not currently available from Enlighten.


Despite the ever-increasing popularity of crowdsourcing (CS) in both industry and academia, procedures that ensure quality in its results are still elusive. We hypothesise that a CS design based on game theory can persuade workers to perform their tasks as quickly as possible with the highest quality. In order to do so, in this article we propose a CS framework inspired by the n-person Chicken game. Our aim is to address the problem of CS quality without compromising on CS benefits such as low monetary cost and high task completion speed. With that goal in mind, we study the effects of knowledge updates as well as incentives for good workers to continue playing. We define a general task with the characteristics of relevance assessment as a case study, because it has been widely explored in the past with CS due to its potential cost and complexity. In order to investigate our hypotheses, we conduct a simulation where we study the effect of the proposed framework on data accuracy, task completion time, and total monetary rewards. Based on a game-theoretical analysis, we study how different types of individuals would behave under a particular game scenario. In particular, we simulate a population comprised of different types of workers with varying ability to formulate optimal strategies and learn from their experiences. A simulation of the proposed framework produced results that support our hypothesis.

Item Type:Articles
Glasgow Author(s) Enlighten ID:Jose, Professor Joemon and Moshfeghi, Dr Yashar
Authors: Moshfeghi, Y., Rosero, A. F. H., and Jose, J.
College/School:College of Science and Engineering > School of Computing Science
Journal Name:ACM Transactions on Intelligent Systems and Technology
Publisher:Association for Computing Machinery
ISSN (Online):2157-6912

University Staff: Request a correction | Enlighten Editors: Update this record