Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error. (arXiv:2201.12417v2 [cs.LG] UPDATED)

In this work, we study the use of the Bellman equation as a surrogate
objective for value prediction accuracy. While the Bellman equation is uniquely
solved by the true value function over all state-action pairs, we find that the
Bellman error (the difference between both sides of the equation) is a poor
proxy for the accuracy of the value function. In particular, we show that (1)
due to cancellations from both sides of the Bellman equation, the magnitude of
the …

arxiv error lg trust value