Saliency: a new selection criterion of important architectures in neural architecture search

Neural architecture search (NAS) has achieved great success in automatically designing high-performance neural networks for given tasks. But the early NAS approaches have a problem of excessive computational cost. Recently, some NAS approaches, such as gradient-based ones, have significantly reduced the computational cost. However, the gradient-based methods have a significant deviation in the architecture selection because they simply use the parameter values of the corresponding architectures as an importance index for architecture selection. This causes the architecture selected from the search space to generally fall into a sub-optimal state. To address this problem, we propose architecture saliency, as a new selection criterion of optimal architectures. Concretely, we define architecture saliency as the squared change in network loss induced by removing this architecture from the neural network. Our saliency directly reflects the contribution of a candidate architecture to the network performance. Therefore, our proposed selection criterion eliminates the deviation in architecture selection. Furthermore, we approximate architecture saliency with Taylor series expansion to get a more efficient implementation. Extensive experiments show that our approach achieves competitive even better model evaluation performance than other NAS approaches on multiple datasets.