Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methods

arXiv:2310.02671v2 Announce Type: replace-cross
Abstract: Markov Decision Processes (MDPs) are a formal framework for modeling and solving sequential decision-making problems. In finite-time horizons such problems are relevant for instance for optimal stopping or specific supply chain problems, but also in the training of large language models. In contrast to infinite horizon MDPs optimal policies are not stationary, policies must be learned for every single epoch. In practice all parameters are often trained simultaneously, ignoring the inherent structure suggested by dynamic …

abstract analysis arxiv beyond contrast convergence cs.lg decision framework gradient instance language language models large language large language models making markov math.oc modeling policy processes softmax stat.ml stochastic supply chain training type