Improving RL with lookahead: learning off-policy with online planning

Overview of LOOP: LOOP reduces dependency on value errors by using an H-step Lookahead Policy that plans online using learned dynamics with a terminal value function. The value function is efficiently learned by a model-free off-policy algorithm using the transitions collected in the environment when the H-step Lookahead Policy is deployed. LOOP is a desirable framework with its strong performance in Online RL, Offline RL, and Safe RL, which is shown in Locomotion, Manipulation, and Navigation environments.

articles deep dive learning policy rl