AFU: Actor-Free critic Updates in off-policy RL for continuous control

arXiv:2404.16159v1 Announce Type: new
Abstract: This paper presents AFU, an off-policy deep RL algorithm addressing in a new way the challenging "max-Q problem" in Q-learning for continuous action spaces, with a solution based on regression and conditional gradient scaling. AFU has an actor but its critic updates are entirely independent from it. As a consequence, the actor can be chosen freely. In the initial version, AFU-alpha, we employ the same stochastic actor as in Soft Actor-Critic (SAC), but we then …

abstract actor algorithm arxiv continuous control cs.ai cs.lg deep rl free gradient independent max paper policy q-learning regression scaling solution spaces type updates