The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization

arXiv:2403.17031v1 Announce Type: new
Abstract: This work is the first to openly reproduce the Reinforcement Learning from Human Feedback (RLHF) scaling behaviors reported in OpenAI's seminal TL;DR summarization work. We create an RLHF pipeline from scratch, enumerate over 20 key implementation details, and share key insights during the reproduction. Our RLHF-trained Pythia models demonstrate significant gains in response quality that scale with model size, with our 2.8B, 6.9B models outperforming OpenAI's released 1.3B checkpoint. We publicly release the trained model …

arxiv case case study cs.lg implementation ppo rlhf study summarization type