FedDIP: Federated Learning with Extreme Dynamic Pruning and Incremental Regularization

Long, Q. , Anagnostopoulos, C. , Puthiya, S. and Bi, D. (2024) FedDIP: Federated Learning with Extreme Dynamic Pruning and Incremental Regularization. In: IEEE ICDM 2023, Shanghai, China, 1-4 December 2023, pp. 1187-1192. ISBN 9798350307887 (doi: 10.1109/ICDM58522.2023.00146)

Abstract

Federated Learning (FL) has been successfully adopted for distributed training and inference of large-scale Deep Neural Networks (DNNs). However, DNNs are characterized by an extremely large number of parameters, thus, yielding significant challenges in exchanging these parameters among distributed nodes and managing the memory. Although recent DNN compression methods (e.g., sparsification, pruning) tackle such challenges, they do not holistically consider an adaptively controlled reduction of parameter exchange while maintaining high accuracy levels. We, therefore, contribute with a novel FL framework (coined FedDIP), which combines (i) dynamic model pruning with error feedback to eliminate redundant information exchange, which contributes to significant performance improvement, with (ii) incremental regularization that can achieve extreme sparsity of models. We provide convergence analysis of FedDIP and report on a comprehensive performance and comparative assessment against state-of-the-art methods using benchmark data sets and DNN models. Our results showcase that FedDIP not only controls the model sparsity but efficiently achieves similar or better performance compared to other model pruning methods adopting incremental regularization during distributed model training. The code is available at: https://github.com/EricLoong/feddip.

• Item Type: Conference Proceedings
• Additional Information: This work is partially funded by the EU Horizon Grant ‘Integration and Harmonization of Logistics Operations’ TRACE (#101104278).
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Anagnostopoulos, Dr Christos and Puthiya Parambath, Dr Sham and LONG, QIANYU
• Authors: Long, Q., Anagnostopoulos, C., Puthiya, S., and Bi, D.
• College/School: College of Science and Engineering > School of Computing Science
• ISSN: 2374-8486
• ISBN: 9798350307887
• Copyright Holders: Copyright © 2023 IEEE
• First Published: First published in 2023 IEEE International Conference on Data Mining (ICDM)
• Publisher Policy: Reproduced in accordance with the publisher copyright policy