PERL: Parameter Efficient Reinforcement Learning from Human Feedback

arXiv:2403.10704v1 Announce Type: new
Abstract: Reinforcement Learning from Human Feedback (RLHF) has proven to be a strong method to align Pretrained Large Language Models (LLMs) with human preferences. But training models with RLHF is computationally expensive, and an overall complex process. In this work, we study RLHF where the underlying models are trained using the parameter efficient method of Low-Rank Adaptation (LoRA) introduced by Hu et al. [2021]. We investigate the setup of "Parameter Efficient Reinforcement Learning" (PERL), in which …

abstract arxiv cs.ai cs.cl cs.lg feedback human human feedback language language models large language large language models llms perl process reinforcement reinforcement learning rlhf study training training models type work