[R][P] KV Cache is huge and bottlenecks LLM inference. We quantize them to 2bit in a finetuning-free + plug-and-play fashion.

It is well known that batch inference is a common practice for efficient LLM serving (which is one primary reason why services like ChatGPT have an initial delay). This batching practice is motivated by the fact that inference latency is mostly limited by the I/O cost of model loading but not the actual compute, where serving multiple requests in a batched manner adds tolerable latency increase while bringing in massive savings on cost per token. However, one issue of batched …

cache challenge explore index key machinelearning observation outliers quantization the key tokens value wise