Randomized K-FACs: Speeding up K-FAC with Randomized Numerical Linear Algebra. (arXiv:2206.15397v2 [cs.LG] UPDATED)

K-FAC is a successful tractable implementation of Natural Gradient for Deep
Learning, which nevertheless suffers from the requirement to compute the
inverse of the Kronecker factors (through an eigen-decomposition). This can be
very time-consuming (or even prohibitive) when these factors are large. In this
paper, we theoretically show that, owing to the exponential-average
construction paradigm of the Kronecker factors that is typically used, their
eigen-spectrum must decay. We show numerically that in practice this decay is
very rapid, leading to …

arxiv linear linear algebra numerical