Random Weight Factorization Improves the Training of Continuous Neural Representations. (arXiv:2210.01274v2 [cs.LG] UPDATED)

Continuous neural representations have recently emerged as a powerful and
flexible alternative to classical discretized representations of signals.
However, training them to capture fine details in multi-scale signals is
difficult and computationally expensive. Here we propose random weight
factorization as a simple drop-in replacement for parameterizing and
initializing conventional linear layers in coordinate-based multi-layer
perceptrons (MLPs) that significantly accelerates and improves their training.
We show how this factorization alters the underlying loss landscape and
effectively enables each neuron in the …

arxiv continuous factorization random training