The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games. (arXiv:2103.01955v4 [cs.LG] UPDATED)

Proximal Policy Optimization (PPO) is a ubiquitous on-policy reinforcement
learning algorithm but is significantly less utilized than off-policy learning
algorithms in multi-agent settings. This is often due to the belief that PPO is
significantly less sample efficient than off-policy methods in multi-agent
systems. In this work, we carefully study the performance of PPO in cooperative
multi-agent settings. We show that PPO-based multi-agent algorithms achieve
surprisingly strong performance in four popular multi-agent testbeds: the
particle-world environments, the StarCraft multi-agent challenge, Google …

arxiv games ppo