Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedback

arXiv:2404.10776v1 Announce Type: new
Abstract: Learning from human feedback plays an important role in aligning generative models, such as large language models (LLM). However, the effectiveness of this approach can be influenced by adversaries, who may intentionally provide misleading preferences to manipulate the output in an undesirable or harmful direction. To tackle this challenge, we study a specific model within this problem domain--contextual dueling bandits with adversarial feedback, where the true preference label can be flipped by an adversary. We …

abstract adversarial algorithms arxiv cs.lg feedback generative generative models however human human feedback language language models large language large language models llm role type