Linearizing Transformer with Key-Value Memory. (arXiv:2203.12644v3 [cs.CL] UPDATED)

Efficient transformer variants with linear time complexity have been
developed to mitigate the quadratic computational overhead of the vanilla
transformer. Among them are low-rank projection methods such as Linformer and
kernel-based Transformers. Despite their unique merits, they usually suffer
from a performance drop comparing with the vanilla transformer on many sequence
generation tasks, and often fail to obtain computation gain when the generation
is short. We propose MemSizer, an approach towards closing the performance gap
while improving the efficiency even …

arxiv memory transformer value