Implementation of Proximal Meta-Policy Search (ProMP) as well as related Meta-RL algorithm. Includes a useful experiment framework for Meta-RL.
Implementations corresponding to ProMP (Rothfuss et al., 2018). Overall this repository consists of two branches:
The code is written in Python 3 and builds on Tensorflow. Many of the provided reinforcement learning environments require the Mujoco physics engine. Overall the code was developed under consideration of modularity and computational efficiency. Many components of the Meta-RL algorithm are parallelized either using either MPI or Tensorflow in order to ensure efficient use of all CPU cores.
An API specification and explanation of the code components can be found here. Also the documentation can be build locally by running the following commands
# ensure that you are in the root folder of the project
cd docs
# install the sphinx documentaiton tool dependencies
pip install requirements.txt
# build the documentaiton
make clean && make html
# now the html documentation can be found under docs/build/html/index.html
The provided code can be either run in A) docker container provided by us or B) using python on your local machine. The latter requires multiple installation steps in order to setup dependencies.
If not installed yet, set up docker on your machine.
Pull our docker container jonasrothfuss/promp
from docker-hub:
docker pull jonasrothfuss/promp
All the necessary dependencies are already installed inside the docker container.
Ensure that you have a working MPI implementation (see here for more instructions).
For Ubuntu you can install MPI through the package manager:
sudo apt-get install libopenmpi-dev
pip install --upgrade virtualenv
virtualenv <venv-name>
source <venv-name>/bin/activate
If not done yet, install anaconda by following the instructions here.
Then reate a anaconda environment, activate it and install the requirements in requirements.txt
.
conda create -n <env-name> python=3.6
source activate <env-name>
pip install -r requirements.txt
For running the majority of the provided Meta-RL environments, the Mujoco physics engine as well as a corresponding python wrapper are required. For setting up Mujoco and mujoco-py, please follow the instructions here.
In order to run the ProMP algorithm point environment (no Mujoco needed) with default configurations execute:
python run_scripts/pro-mp_run_point_mass.py
To run the ProMP algorithm in a Mujoco environment with default configurations:
python run_scripts/pro-mp_run_mujoco.py
The run configuration can be change either in the run script directly or by providing a JSON configuration file with all the necessary hyperparameters. A JSON configuration file can be provided through the flag. Additionally the dump path can be specified through the dump_path flag:
python run_scripts/pro-mp_run.py --config_file <config_file_path> --dump_path <dump_path>
Additionally, in order to run the the gradient-based meta-learning methods MAML and E-MAML (Finn et. al., 2017 and Stadie et. al., 2018) in a Mujoco environment with the default configuration execute, respectively:
python run_scripts/maml_run_mujoco.py
python run_scripts/e-maml_run_mujoco.py
To cite ProMP please use
@article{rothfuss2018promp,
title={ProMP: Proximal Meta-Policy Search},
author={Rothfuss, Jonas and Lee, Dennis and Clavera, Ignasi and Asfour, Tamim and Abbeel, Pieter},
journal={arXiv preprint arXiv:1810.06784},
year={2018}
}
This repository includes environments introduced in (Duan et al., 2016, Finn et al., 2017).