Teaching Large Language Models to Reason with Reinforcement Learning

arXiv:2403.04642v1 Announce Type: new
Abstract: Reinforcement Learning from Human Feedback (\textbf{RLHF}) has emerged as a dominant approach for aligning LLM outputs with human preferences. Inspired by the success of RLHF, we study the performance of multiple algorithms that learn from feedback (Expert Iteration, Proximal Policy Optimization (\textbf{PPO}), Return-Conditioned RL) on improving LLM reasoning capabilities. We investigate both sparse and dense rewards provided to the LLM both heuristically and via a learned reward model. We additionally start from multiple model sizes …

abstract algorithms arxiv cs.lg expert feedback human human feedback iteration language language models large language large language models learn llm multiple optimization performance policy ppo reason reinforcement reinforcement learning rlhf study success teaching type