[R] Random Weight Factorization Improves the Training of Continuous Neural Representations

[https://arxiv.org/abs/2210.01274](https://arxiv.org/abs/2210.01274)

Abstract: Continuous neural representations have recently emerged as a powerful and flexible alternative to classical discretized representations of signals. However, training them to capture fine details in multi-scale signals is difficult and computationally expensive. Here we propose random weight factorization as a simple drop-in replacement for parameterizing and initializing conventional linear layers in coordinate-based multi-layer perceptrons (MLPs) that significantly accelerates and improves their training. We show how this factorization alters the underlying loss landscape and effectively enables each neuron …

continuous factorization machinelearning random training