Leftover-Lunch: Advantage-based Offline Reinforcement Learning for Language Models

arXiv:2305.14718v4 Announce Type: replace
Abstract: Reinforcement Learning with Human Feedback (RLHF) is the most prominent method for Language Model (LM) alignment. However, RLHF is an unstable and data-hungry process that continually requires new high-quality LM-generated data for finetuning. We introduce Advantage-Leftover Lunch RL (A-LoL), a new class of offline policy gradient algorithms that enable RL training on any pre-existing data. By assuming the entire LM output sequence as a single action, A-LoL allows incorporating sequence-level classifiers or human-designed scoring functions …

abstract algorithms alignment arxiv class cs.cl data feedback finetuning generated gradient however human human feedback language language model language models offline policy process quality reinforcement reinforcement learning rlhf type