Finite-Time Error Analysis of Soft Q-Learning: Switching System Approach

arXiv:2403.06366v1 Announce Type: new
Abstract: Soft Q-learning is a variation of Q-learning designed to solve entropy regularized Markov decision problems where an agent aims to maximize the entropy regularized value function. Despite its empirical success, there have been limited theoretical studies of soft Q-learning to date. This paper aims to offer a novel and unified finite-time, control-theoretic analysis of soft Q-learning algorithms. We focus on two types of soft Q-learning algorithms: one utilizing the log-sum-exp operator and the other employing …

abstract agent analysis arxiv cs.lg decision entropy error function markov paper q-learning solve studies success type value variation