Evaluation of Context-Aware Language Models and Experts for Effort Estimation of Software Maintenance Issues

Alhamed, M. and Storer, T. (2022) Evaluation of Context-Aware Language Models and Experts for Effort Estimation of Software Maintenance Issues. In: 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME 2022), Limisol, Cyprus, 03-07 Oct 2022, pp. 129-138. ISBN 9781665479578 (doi: 10.1109/ICSME55016.2022.00020)

Abstract

Reflecting upon recent advances in Natural Language Processing (NLP), this paper evaluates the effectiveness of context-aware NLP models for predicting software task effort estimates. Term Frequency–Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT) were used as feature extraction methods; Random forest and BERT feed-forward linear neural networks were used as classifiers. Using three datasets drawn from open-source projects and one from a commercial project, the paper evaluates the models and compares the best performing model with expert estimates from both kinds of datasets. The results suggest that BERT as a feature extraction and classifier shows slightly better performance than other combinations, but that there is no significant difference between the presented methods. On the other hand, the results show that expert and Machine Learning (ML) estimate performances are similar, with the experts’ performance being slightly better. Both findings confirmed existing literature, but using substantially different experimental settings.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Storer, Dr Tim and Alhamed, Mohammed Abdullah M
• Authors: Alhamed, M., and Storer, T.
• Subjects: Q Science > QA Mathematics > QA76 Computer software
• College/School: College of Science and Engineering > School of Computing Science; College of Science and Engineering
• ISSN: 1063-6773
• ISBN: 9781665479578
• Copyright Holders: Copyright © 2022 IEEE
• First Published: First published in Proceedings of the 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp 129-138
• Publisher Policy: Reproduced in accordance with the copyright policy of the publisher