HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference

arXiv:2402.09360v1 Announce Type: new
Abstract: Autoregressive decoding with generative Large Language Models (LLMs) on accelerators (GPUs/TPUs) is often memory-bound where most of the time is spent on transferring model parameters from high bandwidth memory (HBM) to cache. On the other hand, recent works show that LLMs can maintain quality with significant sparsity/redundancy in the feedforward (FFN) layers by appropriately training the model to operate on a top-$k$ fraction of rows/columns (where $k \approx 0.05$), there by suggesting a way to …

abstract accelerators arxiv bandwidth cache cs.ai cs.lg decoding generative gpus hbm inference language language models large language large language models llm llms memory parameters quality recall show tpus type