Energy-Latency Manipulation of Multi-modal Large Language Models via Verbose Samples

arXiv:2404.16557v1 Announce Type: new
Abstract: Despite the exceptional performance of multi-modal large language models (MLLMs), their deployment requires substantial computational resources. Once malicious users induce high energy consumption and latency time (energy-latency cost), it will exhaust computational resources and harm availability of service. In this paper, we investigate this vulnerability for MLLMs, particularly image-based and video-based ones, and aim to induce high energy-latency cost during inference by crafting an imperceptible perturbation. We find that high energy-latency cost can be manipulated …

abstract arxiv availability computational consumption cost cs.ai cs.cv deployment energy harm language language models large language large language models latency manipulation mllms modal multi-modal paper performance resources samples service type via vulnerability will