Radial Networks: Dynamic Layer Routing for High-Performance Large Language Models

arXiv:2404.04900v1 Announce Type: new
Abstract: Large language models (LLMs) often struggle with strict memory, latency, and power demands. To meet these demands, various forms of dynamic sparsity have been proposed that reduce compute on an input-by-input basis. These methods improve over static methods by exploiting the variance across individual inputs, which has steadily grown with the exponential increase in training data. Yet, the increasing depth within modern models, currently with hundreds of layers, has opened opportunities for dynamic layer sparsity, …

abstract arxiv compute cs.cl dynamic forms inputs language language models large language large language models latency layer llms memory networks performance power reduce routing sparsity struggle type variance