Improving RL with Lookahead: Learning Off-Policy with Online Planning

Overview of LOOP: LOOP reduces dependency on value errors by using an H-step Lookahead Policy that plans online using learned dynamics with a terminal value function. The value function is efficiently learned by a model-free off-policy algorithm using the transitions collected in the environment when the H-step Lookahead Policy is deployed. LOOP is a desirable framework with its strong performance in Online RL, Offline RL, and Safe RL, which is shown in Locomotion, Manipulation, and Navigation environments.

learning machine learning policy reinforcement learning research rl