Sparse and Structured Hopfield Networks

arXiv:2402.13725v1 Announce Type: new
Abstract: Modern Hopfield networks have enjoyed recent interest due to their connection to attention in transformers. Our paper provides a unified framework for sparse Hopfield networks by establishing a link with Fenchel-Young losses. The result is a new family of Hopfield-Fenchel-Young energies whose update rules are end-to-end differentiable sparse transformations. We reveal a connection between loss margins, sparsity, and exact memory retrieval. We further extend this framework to structured Hopfield networks via the SparseMAP transformation, which …

abstract arxiv attention cs.lg differentiable family framework losses modern networks paper rules transformers type update young