Gradient Sparsification For Communication-Efficient Distributed Optimization

Authors:
Jianqiao Wangni University of Pennsylvania
Jialei Wang Two Sigma Investments, University of Chicago
Ji Liu University of Rochester, Tencent AI lab
Tong Zhang Tencent AI Lab

Introduction:

Modern large-scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures.

Abstract:

Modern large-scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures. A key bottleneck is the communication overhead for exchanging information such as stochastic gradients among different workers. In this paper, to reduce the communication cost, we propose a convex optimization formulation to minimize the coding length of stochastic gradients. The key idea is to randomly drop out coordinates of the stochastic gradient vectors and amplify the remaining coordinates appropriately to ensure the sparsified gradient to be unbiased. To solve the optimal sparsification efficiently, several simple and fast algorithms are proposed for an approximate solution, with a theoretical guarantee for sparseness. Experiments on $\ell_2$ regularized logistic regression, support vector machines, and convolutional neural networks validate our sparsification approaches.

You may want to know: