Improving Offline RL by Blending Heuristics

arXiv:2306.00321v2 Announce Type: replace
Abstract: We propose Heuristic Blending (HUBL), a simple performance-improving technique for a broad class of offline RL algorithms based on value bootstrapping. HUBL modifies the Bellman operators used in these algorithms, partially replacing the bootstrapped values with heuristic ones that are estimated with Monte-Carlo returns. For trajectories with higher returns, HUBL relies more on the heuristic values and less on bootstrapping; otherwise, it leans more heavily on bootstrapping. HUBL is very easy to combine with many …

abstract algorithms arxiv bootstrapping class cs.lg heuristics monte-carlo offline operators performance returns simple type value values