Neural Circuit Policies Enabling Auditable Autonomy

pip3 install -U ncps

Docs
PyPI package repo
Open Access Paper

Neural Circuit Policies (NCPs) are designed sparse recurrent neural networks loosely inspired by the nervous system of the organism C. elegans.

@article{lechner2020neural,
  title={Neural circuit policies enabling auditable autonomy},
  author={Lechner, Mathias and Hasani, Ramin and Amini, Alexander and Henzinger, Thomas A and Rus, Daniela and Grosu, Radu},
  journal={Nature Machine Intelligence},
  volume={2},
  number={10},
  pages={642--652},
  year={2020},
  publisher={Nature Publishing Group}
}

Reproducibility materials for the paper Neural Circuit Policies Enabling Auditable Autonomy

This page serves the purpose of documenting the code, materials, and data used for performing the experiments reported in the paper.

Note that the python code for training and evaluating the models has been written over a period of more than two years and a lot of changes have been made during that time. As a result, there are a few caveats with the code:

• The code contains a lot legacy code, which is not used anymore (e.g., some parts of the data augmentation).
• The code is very sparsely documented
• The code is written in TensorFlow 1.X (tested with 1.14)

For a polished, much more user-friendly, TensorFlow 2.x reference implementation of NCPs, we refer to the main project page.

Introduction

The following page describes the ncp_lab_notebook.tar.gz archive that was submitted alongside the paper for the peer-review. To ensure that we do not temper the archive after publication, the SHA-256 sum the file is 0f22b3d7b2986e343e1d3012f51fdf09f876cde2f1f0a6fdb850acded4fc387e. As the full archive is around 200 GB large, we are unable to publicly host all parts. For inquiries requesting the full archive please drop an email.

We are able to publicly host the complete python training code and Matlab analysis code here (87 MB).

Moreover, we are able to publicly host the code as above and the data generated by the active test runs that were analysis by our Matlab scripts here (1.5 GB)

In particlar, both smaller archives linked above do not include the training data (passive and active), as well as the rosbag recordings from the real car.

Note that the Apache License 2.0 of this repository does not apply to the reproducibility materials downloadable by the links above. The copyright of the reproducibility materials belongs to the authors of the paper.

Archive structure

Generally, this archive contains the materials to do the following three things:

1. Train various models on the passive dataset
2. Train various models on the active dataset
3. Analyze the logs of the control of the car by the active models

Not included in the archive is the code stack that runs on the car and is used to collect the training data, as well as deploy the models for controlling the real car.

System description and external libraries used

All models were trained on Ubuntu 16.04 machines using Python3.5 with TensorFlow 1.14.0. Data analysis was performed using Python2.7, Python3.6, and Matlab 2019a.

Directory structure description

This archive is composed of 11 sub-directories:

• training_scripts: Contains the code to train the passive and active models
• active_test_analysis: Contains the code to analyze the logs produced by testing the models on the active steering test
• pretrained_active_models: Pretrained weights of the models tested on the active steering test
• pretrained_passive_models: Pretrained weights of the models tested on the passive evaluation
• training_data_active: Training data for the active steering test
• training_data_passive: Training data for the passive steering test
• active_test_recordings: Logs produced by testing the models on the active steering test
• Lipschitz_analysis: contains code to reproduce the smoothness analysis of the RNN dynamics in active steering test
• Neural_activity_analysis: contains code to project neural activity of different RNNs on the road during active steering test
• PCA_analysis: contains code to compute principle component analysis on RNN's internal dynamics
• SSIM_Crash_analysis: contains code to compute the structural similarity index of saliency maps while input noise variance is increasing.
• analysis_data: data for the Lipschitz, neural activity, PCA, and SSIM analysis
• saliency_widget: HTML visualization to inspect the attention maps of all active test recordings

Auditing the training data

Ideally, we want to have exactly one dataset that we can use for passive evaluation as well as train our model for the active test. However, in our scenario, this is not possible as the roads at our private active test track are from a different probability distribution than the streets observed on public roads. In particular, the roads at our private test track are narrower than standard public roads and furthermore lack any lane markers. Consequently, no model was observed to generalize well from training on data of public roads to an evaluation on the private test track.

As a result, we collected a separate training set by recording a human driving that navigated through the private test track. We train all models for the active test on only the data from the private test track. Our rationale behind this choice is that we have plenty of passive data (from public roads), whereas the private test track is limited in diversity and size. Therefore, training a model on both the data obtained on public roads and the data obtained on the private test track would create an imbalanced towards an excess of public road training data.

All training samples need to be cropped and rescaled before feeding them into any neural network.

The data itself are located in the directories training_data_active and training_data_passive respectively in the form of h5-files.

Auditing the training pipeline

Generally, we want to share as much code as possible for training the active and the passive model in order to have the same conditions in both scenarios. However, due to the difference in objectives in both cases, there are some files in the training pipelines that are not shared. In particular, for the passive evaluation, we perform a 10-fold cross-testing evaluation, whereas, in the active evaluation, we have a single training set and test the model on the real car. Consequently, exactly the data loading scripts, as well as the logging scripts, are the two files that are not shared between the passive and the active training pipeline. The logging code is interleaved with the main training scripts. Therefore we have the following partitioning:

Active test only files:

• active_data_provider.py: Loads the training data for the active test
• train_active_test.py: Main file to train models for the active test

Passive test only files:

• passive_test_data_provider.py: Loads the training data for the passive test and performs splitting for the cross-testing
• run_passive_test.py: Main file to train models for the passive test

Files that are shared between the training pipeline of the passive and active test

• augmentation_utils.py: Code to perform the shadow augmentation and the sample weighting
• perspective_transformation.py: Code to crop and adjust the input images before processing them by the models
• convolution_head.py: Implementation of the convolutional layers that precede each RNN model
• wormflow3.py: Implementation of the NCP model
• models/cnn_model.py: Implementation of the feedforward convolutional neural network model used as a baseline
• models/rnn_models.py: Implementation of the Vanilla RNN, CT-RNN, GRU, and CT-GRU
• models/e2e_lstm.py: Implementation of the LSTM + wrapper to make it compatible to the training pipeline
• models/e2e_worm_pilot.py: Wrapper that makes the NCP model compatible to the training pipeline
• models/e2e_rnn.py: Wrapper that makes various RNNs compatible to the training pipeline

Auditing the model implementation

Implementation of the NCP model, including the ODE solver and architectural design, is implemented in the file wormflow3.py. Furthermore, this module contains methods to export the parameters and hidden states of an NCP model.

Auditing the data analysis code

The logs of the active steering test were collected in the form of rosbag files. The primary module to analyze these rosbags is the file driving_record.py located in the active_test_analysis sub-directory.

Performing the passive evaluation

The main file for running a passive evaluation run is run_passive_test.py. In essence, this module accepts two command-line arguments: --model which defines the type of model and --experiment_id which defines the identifier of the 10-fold cross-testing run. Valid options for the --model argument are

• cnn: Feedforward CNN
• gru: Gated-Recurrent-Unit (GRU) custom implementation
• tf_gru: Gated-Recurrent-Unit (GRU) TensorFlow built-in implementation
• lstm: Long-Short-Term-Memory (LSTM) TensorFlow built-in implementation
• vanilla: Vanilla RNN
• ctrnn: CT-RNN
• ctgru: CT-GRU
• wm: NCP model

Valid option for the --experiment_id argument is a number between 0 and 9 inclusively. For instance,

python3 run_passive_test.py --model cnn --experiment_id 4

Runs the 4-th cross-testing evaluation of the feedforward CNN model.

There are several other command-line options that control parameters like the sparsity level and size of the RNNs, learning rate, dropout rate, and so on. The hyperparameters used are reported in the table in the supplementary materials of the paper.

The script run_passive_test.py creates a directory inside the passive_sessions folder for each model, and inside this directory, a sub-directory for each experiment id. Thus, in total, there will be ten sub-directories within each directory. The python module list_passive_results.py summarizes the results logged in these directories, by averaging over the experiment ids.

The logs created in our experiment runs are located in the pretrained_passive_models directory. To list all results of the passive evaluation run

python3 list_passive_results.py

inside the training_scripts directory.

Training models for the active evaluation

The main file for training a model on the active training set is train_active_test.py. Like with the passive evaluation, this module accepts the command line argument: --model which defines the type of model. Valid values for this parameter are the same as for the passive evaluation script.

For instance

python3 train_active_test.py --model lstm

trains an LSTM model on the active data set. Like with the passive evaluation script, there are other command-line options to tune the hyperparameters.

Each run of this script will create a directory inside active_sessions. Inside this directory, there will be a csv log file that records some training metrics throughout the training procedure. In particular, after every training epoch, the weights will be stored, and training and validation loss recorded. These logs represent the learning curves in the supplementary materials of the paper. The weight-checkpoint used for evaluation on the real car was manually selected by a tradeoff between low validation loss and expected generalization, i.e., earlier epochs with similar validation loss are preferred over slightly lower validation losses after a large number of training epochs.

The training logs and checkpoints created in our training setup are located in the pretrained_active_models directory (one weight-checkpoint for each epoch) The exact weights used for the active steering test are placed in the sub-directory pretrained_active_models/final_models.

Analyzing the active test logs

The active_test_analysis directory contains script to analyze and process the rosbag logs. In particular, the module driving_record.py contains code to count the number of crashes as well as compute secondary metrics such as lateral discomfort. The script export_images.py extracts the camera images, GPS readings, and auto mode switch as separate files in the form of numpy or csv files. These exported files can then be used to re-compute the internal state of the RNNs during driving. The code to do that is located in training_scripts/replay_internal_states.py. The traces of the internal RNN states are then used for the interpretability analysis and the video generation.

Perturbation Analysis

The script located in SSIM_Crash_analysis/ssim_and_crash_plot.m, computes the structural similarity index (SSIM) for the saliency maps when the input noise is increasing. It also plots the number of crashes witnessed during active testing, as a function of input noise variance, for the RNNs. Before running the script, you must:

1. make sure that the content of SSIM_Crash_analysis/saliency_maps is properly included in the path.
2. make sure that the content of SSIM_Crash_analysis/aboxplot is properly included in the path.

Lipschitzness Analysis

The script located in Lipschitz_analysis/lipschitz_plots.m, computes sorted maximum Lipschitz constant for NCPs, CNNs, CT-RNNs, and LSTMs. Data needed for running the scripts is located in the ```analysis_data`` directory.

Principle Component Analysis (PCA)

The script located in PCA_analysis/PCA_explained_and_road_plots.m, performs a PCA on the given RNN network activity during active testing. It outputs the variance explained of principle components and also projects the activity of PC 1 and PC2 on the road. Data needed for running the scripts is located in the ```analysis_data`` directory.

Neural Activity visualization for NCPs, LSTMs, CNNs, and CT-RNNs

The script located in Neural_activity_analysis/plot_data_on_road.m, plots the neural state activity of RNN compartments of the networks on the road on which the car was driven. For the NCP networks, the time-constant (coupling sensitivity) dynamics are also included. Data needed for running the scripts is located in the ```analysis_data`` directory.