Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence

arXiv:2301.11476v4 Announce Type: replace
Abstract: Many policy optimization approaches in reinforcement learning incorporate a Kullback-Leilbler (KL) divergence to the previous policy, to prevent the policy from changing too quickly. This idea was initially proposed in a seminal paper on Conservative Policy Iteration, with approximations given by algorithms like TRPO and Munchausen Value Iteration (MVI). We continue this line of work by investigating a generalized KL divergence -- called the Tsallis KL divergence -- which use the $q$-logarithm in the definition. …

abstract algorithms arxiv cs.ai cs.lg divergence generalized iteration optimization paper policy reinforcement reinforcement learning type value