Preference Ranking Optimization for Human Alignment

arXiv:2306.17492v2 Announce Type: replace
Abstract: Large language models (LLMs) often contain misleading content, emphasizing the need to align them with human values to ensure secure AI systems. Reinforcement learning from human feedback (RLHF) has been employed to achieve this alignment. However, it encompasses two main drawbacks: (1) RLHF exhibits complexity, instability, and sensitivity to hyperparameters in contrast to SFT. (2) Despite massive trial-and-error, multiple sampling is reduced to pair-wise contrast, thus lacking contrasts from a macro perspective. In this paper, …

abstract ai systems alignment arxiv complexity cs.ai cs.cl feedback human human feedback language language models large language large language models llms optimization ranking reinforcement reinforcement learning rlhf secure ai sensitivity systems them type values