Privacy-Aware Semantic Cache for Large Language Models

arXiv:2403.02694v1 Announce Type: new
Abstract: Large Language Models (LLMs) like ChatGPT, Google Bard, Claude, and Llama 2 have revolutionized natural language processing and search engine dynamics. However, these models incur exceptionally high computational costs. For instance, GPT-3 consists of 175 billion parameters and inference on these models also demands billions of floating-point operations. Caching is a natural solution to reduce LLM inference costs on repeated queries. However, existing caching methods are incapable of finding semantic similarities among LLM queries, leading …

abstract arxiv bard billion cache chatgpt claude computational costs cs.ai cs.cl cs.cr cs.dc cs.lg dynamics google google bard gpt gpt-3 inference instance language language models language processing large language large language models llama llama 2 llms natural natural language natural language processing operations parameters privacy processing search search engine semantic type