Paradoxes in numerical comparison of optimization algorithms

Liu, Q., Gehrlein, W. V., Wang, L., Yan, Y., Cao, Y., Chen, W. and Li, Y. (2020) Paradoxes in numerical comparison of optimization algorithms. IEEE Transactions on Evolutionary Computation, 24(4), pp. 777-791. (doi: 10.1109/TEVC.2019.2955110)

Abstract

Numerical comparison is often key to verifying the performance of optimization algorithms, especially, global optimization algorithms. However, studies have so far neglected issues concerning comparison strategies necessary to rank optimization algorithms properly. To fill this gap for the first time, we combine voting theory and numerical comparison research areas, which have been disjoint so far, and thus extend the results of the former to the latter for optimization algorithms. In particular, we investigate compatibility issues arising from comparing two and more than two algorithms, termed “C2” and “C2+” in this article, respectively. Through defining and modeling “C2” and “C2+” mathematically, it is uncovered and illustrated that numerical comparison can be incompatible. Further, two possible paradoxes, namely, “cycle ranking” and “survival of the nonfittest,” are discovered and analyzed rigorously. The occurrence probabilities of these two paradoxes are also calculated under the no-free-lunch assumption, which shows the first justifiable use of the impartial culture assumption from voting theory, providing a point of reference to the frequency of the paradoxes occurring. It is also shown that significant influence on these probabilities comes from the number of algorithms and the number of optimization problems studied in the comparison. Further, various limiting probabilities when the number of optimization problems goes to infinity are also derived and characterized. The results would help guide benchmarking and developing optimization and machine learning algorithms.

• Item Type: Articles
• Additional Information: This work was supported in part by the National Key Research and Development Program of China under Grant 2016YFD0400206, in part by the National Natural Science Fund for Distinguished Young Scholars of China under Grant 61525304, in part by the National Natural Science Foundation of China under Grant 61773119 and Grant 61873328, and in part by the Dongguan University of Technology under Grant KCYKYQD2017014 and Grant DGUT(Q)-GGB-2016005.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Li, Professor Yun
• Authors: Liu, Q., Gehrlein, W. V., Wang, L., Yan, Y., Cao, Y., Chen, W., and Li, Y.
• College/School: College of Science and Engineering > School of Engineering > Systems Power and Energy
• Journal Name: IEEE Transactions on Evolutionary Computation
• Publisher: IEEE
• ISSN: 1089-778X
• ISSN (Online): 1941-0026
• Published Online: 22 November 2019
• Copyright Holders: Copyright © 2019 The Authors
• First Published: First published in IEEE Transactions on Evolutionary Computation 24(4): 777-791
• Publisher Policy: Reproduced under a Creative Commons License