Reflect-RL: Two-Player Online RL Fine-Tuning for LMs

arXiv:2402.12621v1 Announce Type: new
Abstract: As language models (LMs) demonstrate their capabilities in various fields, their application to tasks requiring multi-round interactions has become increasingly popular. These tasks usually have complex dynamics, so supervised fine-tuning (SFT) on a limited offline dataset does not yield good performance. However, only a few works attempted to directly train the LMs within interactive decision-making environments. We aim to create an effective mechanism to fine-tune LMs with online reinforcement learning (RL) in these environments. We …

abstract application arxiv become capabilities cs.cl cs.lg dataset dynamics fields fine-tuning good interactions language language models lms offline performance popular sft supervised fine-tuning tasks type