NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Models

arXiv:2404.01306v1 Announce Type: new
Abstract: Transformer-based Language Models have become ubiquitous in Natural Language Processing (NLP) due to their impressive performance on various tasks. However, expensive training as well as inference remains a significant impediment to their widespread applicability. While enforcing sparsity at various levels of the model architecture has found promise in addressing scaling and efficiency issues, there remains a disconnect between how sparsity affects network topology. Inspired by brain neuronal networks, we explore sparsity approaches through the lens …

abstract algorithm architecture arxiv become cs.cl cs.lg however inference language language models language processing large language large language models natural natural language natural language processing neuro nlp performance processing sparsity tasks training transformer type