Do Differentiable Simulators Give Better Policy Gradients?. (arXiv:2202.00817v2 [cs.LG] UPDATED)

Differentiable simulators promise faster computation time for reinforcement
learning by replacing zeroth-order gradient estimates of a stochastic objective
with an estimate based on first-order gradients. However, it is yet unclear
what factors decide the performance of the two estimators on complex landscapes
that involve long-horizon planning and control on physical systems, despite the
crucial relevance of this question for the utility of differentiable
simulators. We show that characteristics of certain physical systems, such as
stiffness or discontinuities, may compromise the …

arxiv lg policy