Simplified Space Based Neural Architecture Search

In this paper, we propose Simplified Space based Neural Architecture Search (SSNAS), an efficient approach for automatic architecture search. Inspired by the popular convolutional neural networks which have strong capability for feature extraction with small convolutional kernels, we design a simplified search space of convolutional operations with small kernel and construct a large model based on it. Furthermore, we use an Long Short-Term Memory (LSTM) to sample child model from the large model in a way of selective activation. The probability distribution of selective activation is obtained by training the LSTM with reinforcement learning for maximizing the expected reward of selected child model on the validation set. Moreover, the trained weights are saved in a large model which concludes all child models. Each weight of new sampled child model will be restored when being selected by another one again instead of training from scratch so that SSNAS can greatly reduce the expensive computation than traditional approaches for automatic model design. Extensive experiments on CIFAR-10, ImageNet and VOC show that the proposed approach excels in discovering transferable architectures with high performance.