From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification

arXiv:2403.06326v1 Announce Type: cross
Abstract: User alignment is crucial for adapting general-purpose language models (LMs) to downstream tasks, but human annotations are often not available for all types of instructions, especially those with customized constraints. We observe that user instructions typically contain constraints. While assessing response quality in terms of the whole instruction is often costly, efficiently evaluating the satisfaction rate of constraints is feasible. We investigate common constraints in NLP tasks, categorize them into three classes based on the …

abstract alignment annotations arxiv constraints cs.ai cs.cl cs.lg general human language language model language models lms observe quality tasks terms type types verification