Log-sum-exp Optimization based on Continuous Piecewise Linearization Techniques

The log-sum-exp (LSE) function is a class of important non-linear functions that has been widely applied in many fields, such as the signomial geometric optimization, deep neural networks, etc. In this paper, we systematically investigate the log-sum-exp optimization (LSEO) methods. In order to handle the non-linearity and non-convexity of LSEO problems, we propose an iterative algorithm by optimizing a series of continuous piecewise linear (CPWL) problems where LSE functions are approximated using the continuous piecewise linear technique. Since CPWL concave problems have some useful structural properties, we propose an efficient algorithm which solves a series of linear programming problems. To confirm the validity of the proposed method, we test on some toy examples, as well as the multi-class classification based on artificial neural networks with a softmax output layer and the cross-entropy loss function.