CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers

arXiv:2404.06709v1 Announce Type: new
Abstract: The fast-growing large scale language models are delivering unprecedented performance on almost all natural language processing tasks. However, the effectiveness of large language models are reliant on an exponentially increasing number of parameters. The overwhelming computation complexity incurs a high inference latency that negatively affects user experience. Existing methods to improve inference efficiency, such as tensor parallelism and quantization, target to reduce per-layer computing latency, yet overlook the cumulative latency due to the number of …

abstract arxiv complexity computation cs.cl however independent inference inference latency language language models language processing large language large language models latency natural natural language natural language processing optimization parameters performance processing scale tasks type