ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference

arXiv:2312.02554v2 Announce Type: replace
Abstract: Aligning language models to human expectations, e.g., being helpful and harmless, has become a pressing challenge for large language models. A typical alignment procedure consists of supervised fine-tuning and preference learning. Most preference learning methods, such as RLHF and DPO, depend on pairwise preference data, which inadequately address scenarios where human feedback is point-wise, leading to potential information loss and suboptimal performance. Addressing this gap, we introduce Point-wise Direct Preference Optimization, a novel preference learning …

abstract alignment arxiv become challenge cs.cl cs.lg fine-tuning human language language model language models large language large language models rlhf supervised fine-tuning type wise