Model-Advantage and Value-Aware Models for Model-Based Reinforcement Learning: Bridging the Gap in Theory and Practice. (arXiv:2106.14080v2 [cs.LG] UPDATED)

Web: http://arxiv.org/abs/2106.14080

This work shows that value-aware model learning, known for its numerous
theoretical benefits, is also practically viable for solving challenging
continuous control tasks in prevalent model-based reinforcement learning
algorithms. First, we derive a novel value-aware model learning objective by
bounding the model-advantage i.e. model performance difference, between two
MDPs or models given a fixed policy, achieving superior performance to prior
value-aware objectives in most continuous control environments. Second, we
identify the issue of stale value estimates in naively substituting value-aware …

arxiv learning model models reinforcement learning theory value