A Modern Self-Referential Weight Matrix That Learns to Modify Itself. (arXiv:2202.05780v2 [cs.LG] UPDATED)

The weight matrix (WM) of a neural network (NN) is its program. The programs
of many traditional NNs are learned through gradient descent in some error
function, then remain fixed. The WM of a self-referential NN, however, can keep
rapidly modifying all of itself during runtime. In principle, such NNs can
meta-learn to learn, and meta-meta-learn to meta-learn to learn, and so on, in
the sense of recursive self-improvement. While NN architectures potentially
capable of implementing such behaviour have been …

arxiv lg weight matrix