Feedback Loops With Language Models Drive In-Context Reward Hacking

Language models influence the external world: they query APIs that read and write to web pages, generate content that shapes human behavior, and run system commands as autonomous agents. These interactions form feedback loops: LLM outputs affect the world, which in turn affect subsequent LLM outputs. In this work, we show that feedback loops can cause in-context reward hacking (ICRH), where the LLM at test-time optimizes a (potentially implicit) objective but creates negative side effects in the process. For example, …

agents apis autonomous autonomous agents behavior context cs.ai cs.cl cs.lg drive feedback form generate hacking human influence interactions language language models llm query show web work world