REBEL: Reinforcement Learning via Regressing Relative Rewards

arXiv:2404.16767v1 Announce Type: new
Abstract: While originally developed for continuous control problems, Proximal Policy Optimization (PPO) has emerged as the work-horse of a variety of reinforcement learning (RL) applications including the fine-tuning of generative models. Unfortunately, PPO requires multiple heuristics to enable stable convergence (e.g. value networks, clipping) and is notorious for its sensitivity to the precise implementation of these components. In response, we take a step back and ask what a minimalist RL algorithm for the era of generative …

abstract applications arxiv continuous control convergence cs.cl cs.cv cs.lg fine-tuning generative generative models heuristics multiple networks optimization policy ppo reinforcement reinforcement learning type value via while work