Truly No-Regret Learning in Constrained MDPs

arXiv:2402.15776v1 Announce Type: new
Abstract: Constrained Markov decision processes (CMDPs) are a common way to model safety constraints in reinforcement learning. State-of-the-art methods for efficiently solving CMDPs are based on primal-dual algorithms. For these algorithms, all currently known regret bounds allow for error cancellations -- one can compensate for a constraint violation in one round with a strict constraint satisfaction in another. This makes the online learning process unsafe since it only guarantees safety for the final (mixture) policy but …

abstract algorithms art arxiv constraints cs.lg decision error markov primal processes reinforcement reinforcement learning safety state stat.ml type