What Can Linear Interpolation of Neural Network Loss Landscapes Tell Us?

Vlaar, T. J. and Frankle, J. (2022) What Can Linear Interpolation of Neural Network Loss Landscapes Tell Us? In: 39th International Conference on Machine Learning (ICML2022), Baltimore, Maryland, USA, 17-23 July 2022, pp. 22325-22341.

Abstract

Studying neural network loss landscapes provides insights into the nature of the underlying optimization problems. Unfortunately, loss landscapes are notoriously difficult to visualize in a human-comprehensible fashion. One common way to address this problem is to plot linear slices of the landscape, for example from the initial state of the network to the final state after optimization. On the basis of this analysis, prior work has drawn broader conclusions about the difficulty of the optimization problem. In this paper, we put inferences of this kind to the test, systematically evaluating how linear interpolation and final performance vary when altering the data, choice of initialization, and other optimizer and architecture design choices. Further, we use linear interpolation to study the role played by individual layers and substructures of the network. We find that certain layers are more sensitive to the choice of initialization, but that the shape of the linear path is not indicative of the changes in test accuracy of the model. Our results cast doubt on the broader intuition that the presence or absence of barriers when interpolating necessarily relates to the success of optimization.

• Item Type: Conference Proceedings
• Additional Information: During the creation of this paper Tiffany Vlaar was supported by The Maxwell Institute Graduate School in Analysis and its Applications, a Centre for Doctoral Training funded by the UK Engineering and Physical Sciences Research Council (grant EP/L016508/01), the Scottish Funding Council, Heriot-Watt University and the University of Edinburgh
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Vlaar, Dr Tiffany
• Authors: Vlaar, T. J., and Frankle, J.
• College/School: College of Science and Engineering > School of Mathematics and Statistics > Mathematics
• ISSN: 2640-3498
• Copyright Holders: Copyright © 2022 The Author(s)
• First Published: First published in Proceedings of the 39th International Conference on Machine Learning 162:22325-22341
• Publisher Policy: Reproduced under a Creative Commons license