RLVF: Learning from Verbal Feedback without Overgeneralization

arXiv:2402.10893v1 Announce Type: new
Abstract: The diversity of contexts in which large language models (LLMs) are deployed requires the ability to modify or customize default model behaviors to incorporate nuanced requirements and preferences. A convenient interface to specify such model adjustments is high-level verbal feedback, such as "Don't use emojis when drafting emails to my boss." However, while writing high-level feedback is far simpler than collecting annotations for reinforcement learning from human feedback (RLHF), we find that simply prompting a …

abstract arxiv cs.ai cs.cl cs.lg diversity emails feedback language language models large language large language models llms requirements type verbal