Off-Policy Correction for Actor-Critic Methods without Importance Sampling. (arXiv:2208.00755v2 [cs.LG] UPDATED)

Compared to on-policy policy gradient techniques, off-policy model-free deep
reinforcement learning (RL) that uses previously gathered data can improve
sampling efficiency. However, off-policy learning becomes challenging when the
discrepancy between the distributions of the policy of interest and the
policies that collected the data increases. Although the well-studied
importance sampling and off-policy policy gradient techniques were proposed to
compensate for this discrepancy, they usually require a collection of long
trajectories that increases the computational complexity and induce additional
problems such …

actor-critic arxiv importance policy sampling