Outlier-Efficient Hopfield Layers for Large Transformer-Based Models

arXiv:2404.03828v1 Announce Type: new
Abstract: We introduce an Outlier-Efficient Modern Hopfield Model (termed $\mathtt{OutEffHop}$) and use it to address the outlier-induced challenge of quantizing gigantic transformer-based models. Our main contribution is a novel associative memory model facilitating \textit{outlier-efficient} associative memory retrievals. Interestingly, this memory model manifests a model-based interpretation of an outlier-efficient attention mechanism ($\text{Softmax}_1$): it is an approximation of the memory retrieval process of $\mathtt{OutEffHop}$. Methodologically, this allows us to debut novel outlier-efficient Hopfield layers a powerful attention alternative …

abstract arxiv attention challenge cs.ai cs.lg interpretation memory modern novel outlier stat.ml transformer transformer-based models type