The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics

arXiv:2311.09709v2 Announce Type: replace
Abstract: Deploying large language models (LLMs) encounters challenges due to intensive computational and memory requirements. Our research examines vocabulary trimming (VT) inspired by restricting embedding entries to the language of interest to bolster time and memory efficiency. While such modifications have been proven effective in tasks like machine translation, tailoring them to LLMs demands specific modifications given the diverse nature of LLM applications. We apply two language heuristics to trim the full vocabulary - Unicode-based script …

abstract arxiv challenges computational cs.cl efficiency embedding heuristics inference language language model language models large language large language model large language models llms memory requirements research type ups while