Uncovering bugs in code coverage profilers via control flow constraint solving

Wang, Y., Zhang, P., Sun, M., Lu, Z., Yang, Y., Tang, Y. , Qian, J., Li, Z. and Zhou, Y. (2023) Uncovering bugs in code coverage profilers via control flow constraint solving. IEEE Transactions on Software Engineering, 49(11), pp. 4964-4987. (doi: 10.1109/TSE.2023.3321381)

Abstract

Code coverage has been widely used as the basis for various software quality assurance techniques. Therefore, it is of great importance to ensure that coverage profilers provide reliable code coverage. However, it is challenging to validate the correctness of the code coverage generated due to the lack of an effective oracle. In this paper, we propose an effective approach based on control flow constraint solving to test coverage profilers and have implemented a coverage bug hunting tool, DOG (finD cOverage buGs). Our core idea is to leverage inherent control flow features to generate control flow constraints that the resulting coverage statistics should respect. If DOG identifies any unsatisfiable constraints, it signifies the presence of incorrect coverage statistics. In such cases, DOG provides detailed diagnostic information about the suspicious coverage statistics for manual inspection. Compared with the state-of-the-art works, DOG has the following prominent advantages: (1) wide applicability: DOG eliminates the need for multiple coverage profilers (as required by differential testing) and program variants (as needed in metamorphic testing), making it highly versatile; (2) unique testing capability: DOG effectively analyzes and utilizes relationships among available coverage statistics, boosting its testing capabilities; and (3) enhanced interpretability: DOG provides clear control flow explanations for incorrect code coverage, enabling the localization of suspicious coverage areas. During our testing period with DOG, we successfully identified and reported 27 bugs in Gcov and llvm-cov, both widely-used coverage profilers. Of these, 17 bugs have been confirmed (11 have been fixed), 3 were deemed expected behaviors by developers, and 7 remain unresolved. Remarkably, 21 out of 24 unexpected bugs had been latent for over two and a half years, and nearly half of the coverage bugs (10 out of 24) were undetectable by state-of-the-art coverage profiler validators. These results demonstrate the effectiveness and importance of using DOG to improve the reliability of code coverage profilers.

• Item Type: Articles
• Additional Information: This work was supported in part by the Natural Science Foundation of China under Grants 62172205, 62072194, 62162004, 62362006, U21A20474, and 62202306, in part by the Natural Science Foundation of Jiangsu Province under Grant SBK2023022696, in part by the NJU-Huawei Software New Technology Joint Laboratory Fund under Grant TC20230202021-2023-08, and in part by the CCF-Huawei Populus euphratica Fund under Grant CCF-HuaweiSY2022007.
• Keywords: Coverage bugs, control flow, constraint solving, coverage profilers, testing
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Tang, Dr Yutian
• Authors: Wang, Y., Zhang, P., Sun, M., Lu, Z., Yang, Y., Tang, Y., Qian, J., Li, Z., and Zhou, Y.
• College/School: College of Science and Engineering > School of Computing Science
• Journal Name: IEEE Transactions on Software Engineering
• Publisher: IEEE
• ISSN: 0098-5589
• ISSN (Online): 1939-3520
• Published Online: 04 October 2023
• Copyright Holders: Copyright © 2023 IEEE
• First Published: First published in IEEE Transactions on Software Engineering 49(11):4964-4987
• Publisher Policy: Reproduced in accordance with the publisher copyright policy