Evaluating Efficient Performance Estimators of Neural Architectures
2021
Conducting efficient performance estimations of neural architectures is a
major challenge in neural architecture search (NAS). To reduce the architecture
training costs in NAS, one-shot estimators (OSEs) amortize the architecture
training costs by sharing the parameters of one supernet between all
architectures. Recently, zero-shot estimators (ZSEs) that involve no training
are proposed to further reduce the architecture evaluation cost. Despite the
high efficiency of these estimators, the quality of such estimations has not
been thoroughly studied. In this paper, we conduct an extensive and organized
assessment of OSEs and ZSEs on three NAS benchmarks: NAS-Bench-101/201/301.
Specifically, we employ a set of NAS-oriented criteria to study the behavior of
OSEs and ZSEs and reveal that they have certain biases and variances. After
analyzing how and why the OSE estimations are unsatisfying, we explore how to
mitigate the correlation gap of OSEs from several perspectives. For ZSEs, we
find that current ZSEs are not satisfying enough in these benchmark search
spaces, and analyze their biases. Through our analysis, we give out suggestions
for future application and development of efficient architecture performance
estimators. Furthermore, the analysis framework proposed in our work could be
utilized in future research to give a more comprehensive understanding of newly
designed architecture performance estimators. All codes and analysis scripts
are available at https://github.com/walkerning/aw_nas.
Keywords:
- Correction
- Source
- Cite
- Save
- Machine Reading By IdeaReader
0
References
0
Citations
NaN
KQI