DiJiang: Efficient Large Language Models through Compact Kernelization

arXiv:2403.19928v1 Announce Type: cross
Abstract: In an effort to reduce the computational load of Transformers, research on linear attention has gained significant momentum. However, the improvement strategies for attention mechanisms typically necessitate extensive retraining, which is impractical for large language models with a vast array of parameters. In this paper, we present DiJiang, a novel Frequency Domain Kernelization approach that enables the transformation of a pre-trained vanilla Transformer into a linear complexity model with little training costs. By employing a …

arxiv compact cs.cl cs.lg language language models large language large language models through type