Towards smallers, faster decoder-only transformers: Architectural variants and their implications

arXiv:2404.14462v1 Announce Type: new
Abstract: Research on Large Language Models (LLMs) has recently seen exponential growth, largely focused on transformer-based architectures, as introduced by [1] and further advanced by the decoder-only variations in [2]. Contemporary studies typically aim to improve model capabilities by increasing both the architecture's complexity and the volume of training data. However, research exploring how to reduce model sizes while maintaining performance is limited. This study introduces three modifications to the decoder-only transformer architecture: ParallelGPT (p-gpt), LinearlyCompressedGPT …

abstract advanced aim architecture architectures arxiv capabilities complexity cs.lg decoder faster growth language language models large language large language models llms research studies the decoder transformer transformers type variants