Reinforced Self-Training (ReST) for Language Modeling (Paper Explained)

#ai #rlhf #llm

ReST uses a bootsrap-like method to produce its own extended dataset and trains on ever higher-quality subsets of it to improve its own reward. The method allows for re-using the same generated data multiple times and thus has an efficiency advantage with respect to Online RL techniques like PPO.

Paper: https://arxiv.org/abs/2308.08998

Abstract:
Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences. We propose a …

data dataset efficiency explained generated language llm modeling multiple paper ppo quality rest rlhf self-training training trains