Aligning Large Language Models from Self-Reference AI Feedback with one General Principle

arXiv:2406.11190v1 Announce Type: new
Abstract: In aligning large language models (LLMs), utilizing feedback from existing advanced AI rather than humans is an important method to scale supervisory signals. However, it is highly challenging for AI to understand human intentions and societal values, and provide accurate preference feedback based on these. Current AI feedback methods rely on powerful LLMs, carefully designed specific principles to describe human intentions, and are easily influenced by position bias. To address these issues, we propose a …

abstract advanced advanced ai arxiv cs.ai cs.cl feedback general however human humans important language language models large language large language models llms reference scale type values