Learning Associative Memories with Gradient Descent

arXiv:2402.18724v1 Announce Type: new
Abstract: This work focuses on the training dynamics of one associative memory module storing outer products of token embeddings. We reduce this problem to the study of a system of particles, which interact according to properties of the data distribution and correlations between embeddings. Through theory and experiments, we provide several insights. In overparameterized regimes, we obtain logarithmic growth of the ``classification margins.'' Yet, we show that imbalance in token frequencies and memory interferences due to …

abstract arxiv correlations cs.ai cs.lg data distribution dynamics embeddings gradient memories memory products reduce stat.ml study theory through token training type work