Interpolating Between Softmax Policy Gradient and Neural Replicator Dynamics with Capped Implicit Exploration. (arXiv:2206.02036v1 [cs.LG])

Neural replicator dynamics (NeuRD) is an alternative to the foundational
softmax policy gradient (SPG) algorithm motivated by online learning and
evolutionary game theory. The NeuRD expected update is designed to be nearly
identical to that of SPG, however, we show that the Monte Carlo updates differ
in a substantial way: the importance correction accounting for a sampled action
is nullified in the SPG update, but not in the NeuRD update. Naturally, this
causes the NeuRD update to have higher variance …

arxiv dynamics exploration gradient policy