Prompt Highlighter: Interactive Control for Multi-Modal LLMs

arXiv:2312.04302v2 Announce Type: replace
Abstract: This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs) inference: explicit controllable text generation. Multi-modal LLMs empower multi-modality understanding with the capability of semantic generation yet bring less explainability and heavier reliance on prompt contents due to their autoregressive generative nature. While manipulating prompt formats could improve outputs, designing specific and precise prompts per task can be challenging and ineffective. To tackle this issue, we introduce a novel inference method, Prompt Highlighter, which enables …

arxiv control cs.cl cs.cv interactive llms modal multi-modal prompt type