Abstract: Aiming at the  problem of  the performance bottleneck caused by gradient synchronization in data-parallel deep learning tasks, we proposed a dynamic sparse gradient fusion algorithm. The  algorithm synergistically modelled  gradient compression, pipeine techniques, and tensor fusion technology to establish  a theoretical model of the impact of sparse gradient fusion behavior on accuracy. Based on this, the  gradient fusion scheme was found to accelerate gradient synchronization while improving validation accuracy, so as to solve the problem of unstable validation accuracy caused by sparse gradient fusion. Experimental results show that the sparse gradient fusion algorithm reduces communication time by  1.63 times  compared to layer-wise sparsification method, and reduces convergence time by  2.68 times compared to existing sparse gradient fusion algorithms.

Key words: parallel deep learning, gradient sparsification, tensor fusion, communication pipeline technology