Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metrics

arXiv:2303.05101v4 Announce Type: replace
Abstract: Stochastic-gradient sampling methods are often used to perform Bayesian inference on neural networks. It has been observed that the methods in which notions of differential geometry are included tend to have better performances, with the Riemannian metric improving posterior exploration by accounting for the local curvature. However, the existing methods often resort to simple diagonal metrics to remain computationally efficient. This loses some of the gains. We propose two non-diagonal metrics that can be used …

abstract accounting arxiv bayesian bayesian inference cs.lg differential dynamics exploration geometry gradient improving inference metrics networks neural networks performances posterior sampling scalable stat.co stochastic stochastic-gradient type