Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models

arXiv:2404.03827v1 Announce Type: new
Abstract: We propose a two-stage memory retrieval dynamics for modern Hopfield models, termed $\mathtt{U\text{-}Hop}$, with enhanced memory capacity. Our key contribution is a learnable feature map $\Phi$ which transforms the Hopfield energy function into a kernel space. This transformation ensures convergence between the local minima of energy and the fixed points of retrieval dynamics within the kernel space. Consequently, the kernel norm induced by $\Phi$ serves as a novel similarity measure. It utilizes the stored memory …

abstract arxiv capacity convergence cs.ai cs.lg dynamics energy feature function kernel key map memory modern phi retrieval space stage stat.ml text transformation type uniform