Playing Planning Poker in Crowds: Human Computation of Software Effort Estimates

Alhamed, M. and Storer, T. (2021) Playing Planning Poker in Crowds: Human Computation of Software Effort Estimates. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), Madrid, Spain, 22-30 May 2021, pp. 1-12. ISBN 9781665402965 (doi: 10.1109/ICSE43902.2021.00014)

[img] Text
234713.pdf - Accepted Version



Reliable cost effective effort estimation remains a considerable challenge for software projects. Recent work has demonstrated that the popular Planning Poker practice can produce reliable estimates when undertaken within a software team of knowledgeable domain experts. However, the process depends on the availability of experts and can be time-consuming to perform, making it impractical for large scale or open source projects that may curate many thousands of outstanding tasks. This paper reports on a full study to investigate the feasibility of using crowd workers supplied with limited information about a task to provide comparably accurate estimates using Planning Poker. We describe the design of a Crowd Planning Poker (CPP) process implemented on Amazon Mechanical Turk and the results of a substantial set of trials, involving more than 5000 crowd workers and 39 diverse software tasks. Our results show that a carefully organised and selected crowd of workers can produce effort estimates that are of similar accuracy to those of a single expert.

Item Type:Conference Proceedings
Glasgow Author(s) Enlighten ID:Alhamed, Dr Mohammed and Storer, Dr Tim
Authors: Alhamed, M., and Storer, T.
Subjects:Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Q Science > QA Mathematics > QA76 Computer software
College/School:College of Science and Engineering > School of Computing Science
Copyright Holders:Copyright © 2021 The Authors
Publisher Policy:Reproduced in accordance with the publisher copyright policy
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record