Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback

arXiv:2401.11458v2 Announce Type: replace
Abstract: The success of AI assistants based on Language Models (LLMs) hinges on Reinforcement Learning from Human Feedback (RLHF) to comprehend and align with user intentions. However, traditional alignment algorithms, such as PPO, are hampered by complex annotation and training requirements. This reliance limits the applicability of RLHF and hinders the development of professional assistants tailored to diverse human preferences. In this work, we introduce \textit{Linear Alignment}, a novel algorithm that aligns language models with human …

alignment arxiv cs.cl feedback form human linear solution type