When Your AI Deceives You: Challenges with Partial Observability of Human Evaluators in Reward Learning

arXiv:2402.17747v1 Announce Type: new
Abstract: Past analyses of reinforcement learning from human feedback (RLHF) assume that the human fully observes the environment. What happens when human feedback is based only on partial observations? We formally define two failure cases: deception and overjustification. Modeling the human as Boltzmann-rational w.r.t. a belief over trajectories, we prove conditions under which RLHF is guaranteed to result in policies that deceptively inflate their performance, overjustify their behavior to make an impression, or both. To help …

abstract arxiv cases challenges cs.ai cs.lg deception environment failure feedback human human feedback modeling observability reinforcement reinforcement learning rlhf stat.ml the environment type