Learning Useful Representations of Recurrent Neural Network Weight Matrices

arXiv:2403.11998v1 Announce Type: new
Abstract: Recurrent Neural Networks (RNNs) are general-purpose parallel-sequential computers. The program of an RNN is its weight matrix. How to learn useful representations of RNN weights that facilitate RNN analysis as well as downstream tasks? While the mechanistic approach directly looks at some RNN's weights to predict its behavior, the functionalist approach analyzes its overall functionality -- specifically, its input-output mapping. We consider several mechanistic approaches for RNN weights and adapt the permutation equivariant Deep Weight …

abstract analysis arxiv computers cs.lg general how to learn learn matrix network networks neural network neural networks recurrent neural network recurrent neural networks rnn tasks type weight matrix