Prelimit Coupling and Steady-State Convergence of Constant-stepsize Nonsmooth Contractive SA

arXiv:2404.06023v1 Announce Type: cross
Abstract: Motivated by Q-learning, we study nonsmooth contractive stochastic approximation (SA) with constant stepsize. We focus on two important classes of dynamics: 1) nonsmooth contractive SA with additive noise, and 2) synchronous and asynchronous Q-learning, which features both additive and multiplicative noise. For both dynamics, we establish weak convergence of the iterates to a stationary limit distribution in Wasserstein distance. Furthermore, we propose a prelimit coupling technique for establishing steady-state convergence and characterize the limit of …

abstract approximation arxiv asynchronous convergence cs.lg dynamics features focus math.oc math.pr noise q-learning state stat.ml stochastic study type