Cross-Architecture Transfer Learning for Linear-Cost Inference Transformers

arXiv:2404.02684v1 Announce Type: cross
Abstract: Recently, multiple architectures has been proposed to improve the efficiency of the Transformer Language Models through changing the design of the self-attention block to have a linear-cost inference (LCI). A notable approach in this realm is the State-Space Machines (SSMs) architecture, which showed on-par performance on language modeling tasks with the self-attention transformers. However, such an architectural change requires a full pretraining of the weights from scratch, which incurs a huge cost to researchers and …

abstract architecture architectures arxiv attention block cost cs.ai cs.cl cs.lg design efficiency inference language language models linear machines modeling multiple performance realm self-attention space ssms state through transfer transfer learning transformer transformer language models transformers type