Hybrid MPI/OpenMP parallel asynchronous distributed alternating direction method of multipliers

The distributed alternating direction method of multipliers (ADMM) is one of the most widely used algorithms to solve large-scale optimization problems. Since the memory consumption, communication cost and convergence of the distributed ADMM are affected by the number of workers, how to improve the scalability of the distributed ADMM is one of the main challenges. To address this challenge, this paper proposes an asynchronous distributed ADMM based on the hybrid parallel model (HPAD-ADMM), which uses OpenMP for parallelization inside the node and MPI for message passing between nodes in the distributed system. Each worker solves sub-problem in parallel by multithreading, which reduces the system time at each iteration without affecting the convergence of the system or increasing the communication cost and memory consumption. Furthermore, this paper designs efficient parallelized algorithms to solve sub-problems for different applications. For the L1-regularized logistic regression problem, the sub-problem is solved by parallel trust region newton method and system time is reduced by adjusting the accuracy of the sub-problem. For the lasso problem, parallel matrix inversion algorithms are selected dynamically to reduce the system time according to the size of the data set. Finally, large-scale data sets are used to test the performance of the HPAD-ADMM. Experimental results show that compared with the state-of-the-art distributed ADMM, the HPAD-ADMM has higher scalability without losing accuracy.