HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback

arXiv:2403.08309v1 Announce Type: new
Abstract: Reinforcement Learning from AI Feedback (RLAIF) has the advantages of shorter annotation cycles and lower costs over Reinforcement Learning from Human Feedback (RLHF), making it highly efficient during the rapid strategy iteration periods of large language model (LLM) training. Using ChatGPT as a labeler to provide feedback on open-domain prompts in RLAIF training, we observe an increase in human evaluators' preference win ratio for model responses, but a decrease in evaluators' satisfaction rate. Analysis suggests …

abstract advantages annotation arxiv chatgpt costs cs.ai cs.lg domain feedback human human feedback improvements iteration language language model large language large language model llm making reinforcement reinforcement learning rlaif rlhf strategy training type