On the Convergence Rates of Policy Gradient Methods. (arXiv:2201.07443v1 [math.OC])

We consider infinite-horizon discounted Markov decision problems with finite
state and action spaces. We show that with direct parametrization in the policy
space, the weighted value function, although non-convex in general, is both
quasi-convex and quasi-concave. While quasi-convexity helps explain the
convergence of policy gradient methods to global optima, quasi-concavity hints
at their convergence guarantees using arbitrarily large step sizes that are not
dictated by the Lipschitz constant charactering smoothness of the value
function. In particular, we show that when …

arxiv gradient math policy