Conservative DDPG -- Pessimistic RL without Ensemble

arXiv:2403.05732v1 Announce Type: cross
Abstract: DDPG is hindered by the overestimation bias problem, wherein its $Q$-estimates tend to overstate the actual $Q$-values. Traditional solutions to this bias involve ensemble-based methods, which require significant computational resources, or complex log-policy-based approaches, which are difficult to understand and implement. In contrast, we propose a straightforward solution using a $Q$-target and incorporating a behavioral cloning (BC) loss penalty. This solution, acting as an uncertainty measure, can be easily implemented with minimal code and without …

abstract arxiv bias computational contrast cs.ai cs.lg ddpg ensemble policy resources solution solutions type values