SALMON: Self-Alignment with Instructable Reward Models

arXiv:2310.05910v2 Announce Type: replace-cross
Abstract: Supervised Fine-Tuning (SFT) on response demonstrations combined with Reinforcement Learning from Human Feedback (RLHF) constitutes a powerful paradigm for aligning LLM-based AI agents. However, a significant limitation of such an approach is its dependency on high-quality human annotations, making its application to intricate tasks challenging due to difficulties in obtaining consistent response demonstrations and in-distribution response preferences. This paper presents a novel approach, namely SALMON, to align base language models with minimal human supervision, using …

abstract agents ai agents alignment annotations application arxiv cs.ai cs.cl cs.lg feedback fine-tuning however human human feedback llm llm-based ai agents making paradigm quality reinforcement reinforcement learning rlhf sft supervised fine-tuning tasks type