AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition

arXiv:2402.11452v1 Announce Type: new
Abstract: Recent advancements in large language models (LLMs) have shown promise in multi-step reasoning tasks, yet their reliance on extensive manual labeling to provide procedural feedback remains a significant impediment. To address this challenge, in this paper, we propose a novel self-supervised framework AutoPRM that efficiently enhances the fine-tuning of LLMs for intricate reasoning challenges. Specifically, AutoPRM first decomposes complex problems into more manageable subquestions with a controllable granularity switch, then sequentially apply reinforcement learning to …

abstract arxiv challenge cs.cl feedback framework labeling language language models large language large language models llms novel paper question reasoning reliance supervision tasks type via