COPR: Continual Learning Human Preference through Optimal Policy Regularization

arXiv:2310.15694v5 Announce Type: replace
Abstract: The technique of Reinforcement Learning from Human Feedback (RLHF) is a commonly employed method to improve pre-trained Language Models (LM), enhancing their ability to conform to human preferences. Nevertheless, the current RLHF-based LMs necessitate full retraining each time novel queries or feedback are introduced, which becomes a challenging task because human preferences can vary between different domains or tasks. Retraining LMs poses practical difficulties in many real-world situations due to the significant time and computational …

abstract arxiv continual cs.cl cs.lg current feedback human human feedback language language models lms novel policy queries regularization reinforcement reinforcement learning retraining rlhf through type