Off-Policy Primal-Dual Safe Reinforcement Learning

arXiv:2401.14758v2 Announce Type: replace
Abstract: Primal-dual safe RL methods commonly perform iterations between the primal update of the policy and the dual update of the Lagrange Multiplier. Such a training paradigm is highly susceptible to the error in cumulative cost estimation since this estimation serves as the key bond connecting the primal and dual update processes. We show that this problem causes significant underestimation of cost when using off-policy methods, leading to the failure to satisfy the safety constraint. To …

arxiv cs.lg policy primal reinforcement reinforcement learning safe type