HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models

arXiv:2403.13447v1 Announce Type: cross
Abstract: Recent advancements indicate that scaling up Multimodal Large Language Models (MLLMs) effectively enhances performance on downstream multimodal tasks. The prevailing MLLM paradigm, \emph{e.g.}, LLaVA, transforms visual features into text-like tokens using a \emph{static} vision-language mapper, thereby enabling \emph{static} LLMs to develop the capability to comprehend visual information through visual instruction tuning. Although promising, the \emph{static} tuning strategy~\footnote{The static tuning refers to the trained model with static parameters.} that shares the same parameters may constrain performance …

abstract arxiv capability cs.ai cs.cl cs.cv dynamic enabling expert features language language models large language large language models llava llms mllm mllms multimodal paradigm performance scaling scaling up tasks text tokens type vision visual