NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention

arXiv:2403.01273v1 Announce Type: new
Abstract: Large language model inference on Central Processing Units (CPU) is challenging due to the vast quantities of expensive Multiply-Add (MAD) matrix operations in the attention computations. In this paper, we argue that there is a rare gem in modern CPUs, Single-Instruction-Multiple-Data (SIMD) registers, which allow for ultra-low-latency lookups in batch. We leverage this unique capability of CPUs to propose NoMAD-Attention, an efficient attention algorithm that replaces MAD operations with in-register lookups. Through hardware-aware algorithmic designs, …

arxiv attention cpus cs.ai cs.cl cs.lg free inference llm through type