Policy-Guided Diffusion

arXiv:2404.06356v1 Announce Type: new
Abstract: In many real-world settings, agents must learn from an offline dataset gathered by some prior behavior policy. Such a setting naturally leads to distribution shift between the behavior policy and the target policy being trained - requiring policy conservatism to avoid instability and overestimation bias. Autoregressive world models offer a different solution to this by generating synthetic, on-policy experience. However, in practice, model rollouts must be severely truncated to avoid compounding error. As an alternative, …

abstract agents arxiv behavior bias cs.ai cs.lg cs.ro dataset diffusion distribution leads learn offline policy prior shift solution type world world models