Encodings for neural architecture search
Note: this repository has been combined with other naszilla projects into naszilla/naszilla. This repo is deprecated and not maintained. Please use naszilla/naszilla, which has more functionality.
A Study on Encodings for Neural Architecture Search
Colin White, Willie Neiswanger, Sam Nolen, and Yash Savani.
arxiv:2007.04965.
Many algorithms for neural architecture search (NAS) represent each neural architecture in the search space as a directed acyclic graph (DAG), and then search over all DAGs by encoding the adjacency matrix and list of operations as a set of hyperparameters. Recent work has demonstrated that even small changes to the way each architecture is encoded can have a significant effect on the performance of NAS algorithms. We present the first formal study on the effect of architecture encodings for NAS.
If you run experiments on the DARTS search space, you will need our fork of the DARTS repo:
nasbench_only108.tfrecord
in the top level folder of this repoSome of the path-based encoding methods require a hash map from path indices to cell architectures. We have created a pickle file which contains this hash map (size 57MB), located here. Place it in the top level folder of this repo.
meta_neuralnet.ipynb
to train a neural predictor with different encodingsnotebooks/test_nas.ipynb
to test out each algorithm + encoding combinationpython run_experiments_sequential.py --algo_params evo_encodings
This command will run evolutionary search with six different encodings. To run other experiments, open up params.py
.
To run experiments with NAS-Bench-201, download NAS-Bench-201-v1_0-e61699.pth
from here and place it in the top level folder of this repo. Choose between cifar10, cifar100, and imagenet. For example,
python run_experiments_sequential.py --algo_params evo_encodings --search_space nasbench_201_cifar10
Please cite our paper if you use code from this repo:
@inproceedings{white2020study,
title={A Study on Encodings for Neural Architecture Search},
author={White, Colin and Neiswanger, Willie and Nolen, Sam and Savani, Yash},
booktitle={Advances in Neural Information Processing Systems},
year={2020}
}