IterAlign: Iterative Constitutional Alignment of Large Language Models

arXiv:2403.18341v1 Announce Type: new
Abstract: With the rapid development of large language models (LLMs), aligning LLMs with human values and societal norms to ensure their reliability and safety has become crucial. Reinforcement learning with human feedback (RLHF) and Constitutional AI (CAI) have been proposed for LLM alignment. However, these methods require either heavy human annotations or explicitly pre-defined constitutions, which are labor-intensive and resource-consuming. To overcome these drawbacks, we study constitution-based LLM alignment and propose a data-driven constitution discovery and …

abstract alignment arxiv become cai constitutional ai cs.cl development feedback however human human feedback iterative language language models large language large language models llm llms reinforcement reinforcement learning reliability rlhf safety type values