Nonparametric Modern Hopfield Models

arXiv:2404.03900v1 Announce Type: cross
Abstract: We present a nonparametric construction for deep learning compatible modern Hopfield models and utilize this framework to debut an efficient variant. Our key contribution stems from interpreting the memory storage and retrieval processes in modern Hopfield models as a nonparametric regression problem subject to a set of query-memory pairs. Crucially, our framework not only recovers the known results from the original dense modern Hopfield model but also fills the void in the literature regarding efficient …

abstract arxiv construction cs.ai cs.lg cs.ne deep learning framework key memory modern processes query regression retrieval set stat.ml storage type