Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations

We study the robustness of deep reinforcement learning agents when their state observations (e.g., measurements of environment states) contain noises or even adversarial perturbations. For example, a self-driving car can observe its location through GPS, however GPS signal contains uncertainty and the driving policy must take this uncertainty into consideration to plan routes safely. To guarantee performance under even the worst case uncertainty, we study the foundamental state-adversarial Markov Decision Process (SA-MDP) and propose theoretically principled robustness regularizers for PPO, DDPG and DQN. In addition, we also propose two strong adversarial attacks for PPO and DDPG, the maximal action difference (MAD) attack and the robust sarsa (RS) attack. More details can be found in our paper:

Huan Zhang*, Hongge Chen*, Chaowei Xiao, Mingyan Liu, Bo Li, Duane Boning, and Cho-Jui Hsieh, "Robust Deep Reinforcement Learning against Adversarial Perturbations on Observations" (*Equal contribution)
NeurIPS 2020 (Spotlight)

Please also checkout our new work on optimal adversary and alternating training of learned adversary and agent (ATLA) for a robust RL agent:

Huan Zhang*, Hongge Chen*, Duane Boning, and Cho-Jui Hsieh, "Robust Reinforcement Learning on State Observations with Learned Optimal Adversary" (*Equal contribution)
ICLR 2020 (Code)

Robust Deep Reinforcement Learning Demos (SA-DQN, SA-DDPG and SA-PPO)

Pong, Vanilla DQN reward under attack: -21 (trained agent: right paddle)	Pong, SA-DQN reward under attack: 21 (trained agent: right paddle)	RoadRunner, Vanilla DQN reward under attack: 0	RoadRunner, SA-DQN reward under attack: 49900
Humanoid Vanilla PPO reward under attack: 615	Humanoid SA-PPO reward under attack: 6042	Ant Vanilla DDPG reward under attack: 189	Ant SA-DDPG reward under attack: 2025

Code

In our paper, we conduct experiments on deep Q network (DQN) for discrete action space environments (e.g. Atari games), deep deterministic policy gradient (DDPG) for continous action space environments with deterministic actions, and proximal policy optimization (PPO) for continous action space environments with stochastic actions. Our proposed algorithms (SA-PPO, SA-DDPG and SA-PPO) are evaluated using 11 environments.

Reference implementation for SA-DQN can be found at https://github.com/chenhongge/SA_DQN.

Reference implementation for SA-PPO can be found at https://github.com/huanzhang12/SA_PPO.

Reference implementation for SA-DDPG can be found at https://github.com/huanzhang12/SA_DDPG.

Pretrained agents performance

Pretrained DQN agents

See our SA-DQN repository for more details.

Pretrained PPO agents

We repeatedly train each agent configuration at least 15 times, and rank them with their average cumulative rewards over 50 episodes under the strongest attack (among 5 attacks used). We report the performance for agents with median robustnes (we do not cherry-pick the best agents).

See our SA-PPO repository for more details.

Pretrained DDPG agents

We attack each agent with 5 different attacks (random attack, critic attack, MAD attack, RS attack and RS+MAD attack). Here we report the lowest reward of all 5 attacks in "Strongest attack" rows. Additionally, we train each setting 11 times and we report the agent with median robustness (we do not cherry-pick the best results). This is important due to the potential large training variance in RL.

See our SA-DDPG repository for more details.