Head-wise Shareable Attention for Large Language Models

arXiv:2402.11819v1 Announce Type: new
Abstract: Large Language Models (LLMs) suffer from huge number of parameters, which restricts their deployment on edge devices. Weight sharing is one promising solution that encourages weight reuse, effectively reducing memory usage with less performance drop. However, current weight sharing techniques primarily focus on small-scale models like BERT and employ coarse-grained sharing rules, e.g., layer-wise. This becomes limiting given the prevalence of LLMs and sharing an entire layer or block obviously diminishes the flexibility of weight …

abstract arxiv attention bert cs.cl current deployment devices edge edge devices focus head language language models large language large language models llms memory parameters performance scale small solution type usage wise