AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

arXiv:2306.00978v3 Announce Type: replace
Abstract: Large language models (LLMs) have fundamentally transformed the capabilities of numerous applications, from natural language processing to more intricate domain-specific tasks in robotics and autonomous driving. Moreover, the importance of on-device LLMs has grown significantly in the recent years. Running LLMs on edge devices not only promises reduced latency and improved user experience but also aligns with the increasing need for user privacy, as data processing can occur locally. However, the astronomical model sizes of …

abstract applications arxiv autonomous autonomous driving capabilities compression cs.cl devices domain driving edge edge devices importance language language models language processing large language large language models llm llms natural natural language natural language processing processing quantization robotics running specific tasks tasks type