A Case for Validation Buffer in Pessimistic Actor-Critic

arXiv:2403.01014v1 Announce Type: new
Abstract: In this paper, we investigate the issue of error accumulation in critic networks updated via pessimistic temporal difference objectives. We show that the critic approximation error can be approximated via a recursive fixed-point model similar to that of the Bellman value. We use such recursive definition to retrieve the conditions under which the pessimistic critic is unbiased. Building on these insights, we propose Validation Pessimism Learning (VPL) algorithm. VPL uses a small validation buffer to …

abstract actor actor-critic approximation arxiv case cs.lg definition difference error fixed-point issue networks paper recursive show temporal type validation value via