Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization

arXiv:2402.15473v1 Announce Type: cross
Abstract: Reinforcement Learning from Human Feedback (RLHF) has become a dominating strategy in steering Language Models (LMs) towards human values/goals. The key to the strategy is employing a reward model ({$\varphi$}) which can reflect a latent reward model with humans. While this strategy has proven to be effective, the training methodology requires a lot of human preference annotation (usually of the order of tens of thousands) to train {$\varphi$}. Such large-scale preference annotations can be achievable …

abstract arxiv become case commerce cs.cl cs.lg domain domain knowledge e-commerce feedback human human feedback humans key knowledge language language models lms modelling opinion reinforcement reinforcement learning reward model rlhf strategy study summarization the key type values