Model-Based Active Exploration (MAX)

Code for reproducing experiments in Model-Based Active Exploration, ICML 2019

Written in PyTorch v1.0.

Code relies on sacred for managing experiments and hyper-parameters.

Overview:

• envs/: contains the environments used.
• main.py: contains the main algorithm and baselines through modes.
• models.py: a fast parallel implementation of an ensemble of models which can are trained with negative log-likelihood loss.
• utilities.py: contains the all the utilities (exploration objectives) used in the paper.
• imagination.py: contains code that constructs a virtual MDP using the model ensemble.
• sac.py: contains a simple Soft Actor-Critic implementation.
• sacred_fetcher.py: script to download experiment artifacts stored in MongoDB.

Installation

• Install required dependencies:

  sudo apt install libosmesa6-dev libgl1-mesa-glx libglfw3 patchelf
  

• Create conda environment with required dependencies:

  conda env create -f conda_env.yml
  

• Download and setup MuJoCo binaries. The project uses mujoco and mujoco_py version 1.50.

  mkdir ~/.mujoco/
  cd .mujoco/
  wget -c https://www.roboti.us/download/mjpro150_linux.zip
  unzip mjpro150_linux.zip
  rm mjpro150_linux.zip
  

  Obtain MuJoCo license key and place it .mujoco/ directory created above with filename mjkey.txt.

• Append the following to ~/.bashrc:

  # MuJoCo
  export LD_LIBRARY_PATH=:/home/<USER>/.mujoco/mjpro150/bin
  
  if [ -f /usr/lib/x86_64-linux-gnu/libGLEW.so ]; then    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/<USER>/.mujoco/mjpro150/bin:/usr/lib/nvidia-390
      export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libGLEW.so
      export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/nvidia-375
  fi
  
  

• Quick test of MuJoCo installation

  >>> import gym
  >>> gym.make('HalfCheetah-v2')
  

Commands

Execute the commands listed below from the code directory to reproduce the results.

Half Cheetah

• MAX:

python main.py with max_explore env_noise_stdev=0.02

• Trajectory Variance Active Exploration:

python main.py with max_explore utility_measure=traj_stdev policy_explore_alpha=0.2 env_noise_stdev=0.02

• Renyi Divergence Reactive Exploration:

python main.py with max_explore exploration_mode=reactive env_noise_stdev=0.02

• Prediction Error Reactive Exploration:

python main.py with max_explore exploration_mode=reactive utility_measure=pred_err policy_explore_alpha=0.2 env_noise_stdev=0.02

• Random Exploration:

python main.py with random_explore env_noise_stdev=0.02

Ant

• MAX:

python main.py with max_explore env_name=MagellanAnt-v2 env_noise_stdev=0.02 eval_freq=1500 checkpoint_frequency=1500 ant_coverage=True

• Trajectory Variance Active Exploration:

python main.py with max_explore env_name=MagellanAnt-v2 utility_measure=traj_stdev policy_explore_alpha=0.2 env_noise_stdev=0.02 eval_freq=1500 checkpoint_frequency=1500 ant_coverage=True

• Renyi Divergence Reactive Exploration:

python main.py with max_explore env_name=MagellanAnt-v2 exploration_mode=reactive env_noise_stdev=0.02 eval_freq=1500 checkpoint_frequency=1500 ant_coverage=True

• Prediction Error Reactive Exploration:

python main.py with max_explore env_name=MagellanAnt-v2 exploration_mode=reactive utility_measure=pred_err policy_explore_alpha=0.2 env_noise_stdev=0.02 eval_freq=1500 checkpoint_frequency=1500 ant_coverage=True

• Random Exploration:

python main.py with random_explore env_name=MagellanAnt-v2 env_noise_stdev=0.02 eval_freq=1500 checkpoint_frequency=1500 ant_coverage=True

Magellan

Magellan is the internal code name of the project inspired by life of Ferdinand Magellan.