Auto-Encoding Morph-Tokens for Multimodal LLM

arXiv:2405.01926v1 Announce Type: new
Abstract: For multimodal LLMs, the synergy of visual comprehension (textual output) and generation (visual output) presents an ongoing challenge. This is due to a conflicting objective: for comprehension, an MLLM needs to abstract the visuals; for generation, it needs to preserve the visuals as much as possible. Thus, the objective is a dilemma for visual-tokens. To resolve the conflict, we propose encoding images into morph-tokens to serve a dual purpose: for comprehension, they act as visual …

arxiv auto cs.cv encoding llm multimodal tokens type