Learning to Constrain Policy Optimization with Virtual Trust Region. (arXiv:2204.09315v2 [cs.LG] UPDATED)

We introduce a constrained optimization method for policy gradient
reinforcement learning, which uses a virtual trust region to regulate each
policy update. In addition to using the proximity of one single old policy as
the normal trust region, we propose forming a second trust region through
another virtual policy representing a wide range of past policies. We then
enforce the new policy to stay closer to the virtual policy, which is
beneficial if the old policy performs poorly. More importantly, …

arxiv optimization policy trust virtual