Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models

arXiv:2312.06585v4 Announce Type: replace
Abstract: Fine-tuning language models~(LMs) on human-generated data remains a prevalent practice. However, the performance of such models is often limited by the quantity and diversity of high-quality human data. In this paper, we explore whether we can go beyond human data on tasks where we have access to scalar feedback, for example, on math problems where one can verify correctness. To do so, we investigate a simple self-training method based on expectation-maximization, which we call ReST$^{EM}$, …

abstract arxiv beyond cs.lg data diversity explore fine-tuning generated however human language language models lms paper performance practice problem-solving quality scaling self-training tasks training type