Enhancing LLM Safety via Constrained Direct Preference Optimization

arXiv:2403.02475v1 Announce Type: new
Abstract: The rapidly increasing capabilities of large language models (LLMs) raise an urgent need to align AI systems with diverse human preferences to simultaneously enhance their usefulness and safety, despite the often conflicting nature of these goals. To address this important problem, a promising approach is to enforce a safety constraint at the fine-tuning stage through a constrained Reinforcement Learning from Human Feedback (RLHF) framework. This approach, however, is computationally expensive and often unstable. In this …

abstract ai systems arxiv capabilities cs.cl cs.lg direct preference optimization diverse human language language models large language large language models llm llms nature optimization raise safety systems type via