On Speculative Decoding for Multimodal Large Language Models

arXiv:2404.08856v1 Announce Type: cross
Abstract: Inference with Multimodal Large Language Models (MLLMs) is slow due to their large-language-model backbone which suffers from memory bandwidth bottleneck and generates tokens auto-regressively. In this paper, we explore the application of speculative decoding to enhance the inference efficiency of MLLMs, specifically the LLaVA 7B model. We show that a language-only model can serve as a good draft model for speculative decoding with LLaVA 7B, bypassing the need for image tokens and their associated processing …

abstract application arxiv auto bandwidth cs.ai cs.cl cs.lg decoding efficiency explore inference language language models large language large language models llava memory mllms multimodal paper tokens type