Porcupine Neural Networks: Approximating Neural Network Landscapes

Authors:
Soheil Feizi University of Maryland, College Park
Hamid Javadi Rice University
Jesse Zhang Stanford University
David Tse Stanford University

Introduction:

Neural networks have been used prominently in several machine learning and statistics applications.In particular, for two-layer neural networks the authors introduce Porcupine Neural Networks (PNNs) whose weight vectors are constrained to lie over a finite set of lines.

Abstract:

Neural networks have been used prominently in several machine learning and statistics applications. In general, the underlying optimization of neural networks is non-convex which makes analyzing their performance challenging. In this paper, we take another approach to this problem by constraining the network such that the corresponding optimization landscape has good theoretical properties without significantly compromising performance. In particular, for two-layer neural networks we introduce Porcupine Neural Networks (PNNs) whose weight vectors are constrained to lie over a finite set of lines. We show that most local optima of PNN optimizations are global while we have a characterization of regions where bad local optimizers may exist. Moreover, our theoretical and empirical results suggest that an unconstrained neural network can be approximated using a polynomially-large PNN.

You may want to know: