Spartan: a sparsity-adaptive framework to accelerate deep neural network training on GPUs

Dong, S., Sun, Y., Bohm Agostini, N., Karimi, E., Lowell, D., Zhou, J., Cano, J. , Abellán, J. L. and Kaeli, D. (2021) Spartan: a sparsity-adaptive framework to accelerate deep neural network training on GPUs. IEEE Transactions on Parallel and Distributed Systems, 32(10), pp. 2448-2463. (doi: 10.1109/TPDS.2021.3067825)

Abstract

Deep Neural Networks (DNNs) have emerged as an important class of machine learning algorithms, providing accurate solutions to a broad range of applications. Sparsity in activation maps in DNN training presents an opportunity to reduce computations. However, exploiting activation sparsity presents two major challenges: i) profiling activation sparsity during training comes with significant overhead due to computing the degree of sparsity and the data movement; ii) the dynamic nature of activation maps requires dynamic dense-to-sparse conversion during training, leading to significant overhead. In this paper, we present Spartan, a lightweight hardware/software framework to accelerate DNN training on a GPU. Spartan provides a cost effective and programmer-transparent microarchitectural solution to exploit activation sparsity detected during training. Spartan provides an efficient sparsity monitor, a tile-based sparse GEMM algorithm, and a novel compaction engine designed for GPU workloads. Spartan can reduce sparsity profiling overhead by 52.5× on average. For the most compute-intensive layers, i.e., convolutional layers, we can speedup AlexNet by 3.4×, VGGNet-16 by 2.14×, and ResNet-18 by 2.02×, when training on the ImageNet dataset.

• Item Type: Articles
• Keywords: DNN, sparsity, GPU.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Cano Reyes, Dr Jose
• Authors: Dong, S., Sun, Y., Bohm Agostini, N., Karimi, E., Lowell, D., Zhou, J., Cano, J., Abellán, J. L., and Kaeli, D.
• College/School: College of Science and Engineering > School of Computing Science
• Journal Name: IEEE Transactions on Parallel and Distributed Systems
• Journal Abbr.: TPDS
• Publisher: IEEE
• ISSN: 1045-9219
• ISSN (Online): 1558-2183
• Published Online: 22 March 2021
• Copyright Holders: Copyright © 2021 IEEE
• First Published: First published in IEEE Transactions on Parallel and Distributed Systems 32(10):2448-2463
• Publisher Policy: Reproduced in accordance with the copyright policy of the publisher