Score Regularized Policy Optimization through Diffusion Behavior

arXiv:2310.07297v3 Announce Type: replace
Abstract: Recent developments in offline reinforcement learning have uncovered the immense potential of diffusion modeling, which excels at representing heterogeneous behavior policies. However, sampling from diffusion policies is considerably slow because it necessitates tens to hundreds of iterative inference steps for one action. To address this issue, we propose to extract an efficient deterministic inference policy from critic models and pretrained diffusion behavior models, leveraging the latter to directly regularize the policy gradient with the behavior …

abstract arxiv behavior cs.lg diffusion diffusion modeling however inference issue iterative modeling offline optimization policy reinforcement reinforcement learning sampling through type