Weak-to-Strong Extrapolation Expedites Alignment

arXiv:2404.16792v1 Announce Type: new
Abstract: Although the capabilities of large language models (LLMs) ideally scale up with increasing data and compute, they are inevitably constrained by limited resources in reality. Suppose we have a moderately trained LLM (e.g., trained to align with human preference) in hand, can we further exploit its potential and cheaply acquire a stronger model? In this paper, we propose a simple method called ExPO to boost LLMs' alignment with human preference. ExPO assumes that a medium-aligned …

abstract alignment arxiv capabilities compute cs.ai cs.cl cs.lg data exploit human language language models large language large language models llm llms reality resources scale type