Orthogonal Matching Pursuit for Multilayer Perceptions Neural Networks Model Reduction

Neural networks have drawn much attention in modern machine learning community as they have achieved many successful applications, such as image recognition, speech recognition and system identification. According to the principle of parsimony, simpler neural models are preferable to more complex ones if they have similar generalization performance. However, when building a neural networks model, the neuron number is often determined randomly or by trial-and-error. These methods can often lead to the over-complex networks with many redundant neurons and therefore may result in over-fitting problems. In this paper, a new approach is proposed for obtaining a simplified neural networks with fewer neurons but still keeping a good performance comparing to the initial fully networks. More specifically, the initial neural model with a fixed model size is built using Matlab toolbox. Then, the orthogonal matching pursuit method is employed to select important neurons and drop out redundant neurons, leading to a more compact model with reduced size. Two simulation examples are used to demonstrate the effectiveness of the proposed method.