Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation

arXiv:2401.07382v2 Announce Type: replace
Abstract: Reinforcement learning (RL) can align language models with non-differentiable reward signals, such as human preferences. However, a major challenge arises from the sparsity of these reward signals - typically, there is only a single reward for an entire output. This sparsity of rewards can lead to inefficient and unstable learning. To address this challenge, our paper introduces an novel framework that utilizes the critique capability of Large Language Models (LLMs) to produce intermediate-step rewards during …

abstract arxiv beyond challenge critique cs.ai cs.cl differentiable human language language model language models major reinforcement reinforcement learning sparsity text text generation type