DeepStack: Enhancing Multimodal Models with Layered Visual Token Integration for Superior High-Resolution Performance

Instead of feeding a long sequence of visual tokens into the language model’s first layer, DeepStack distributes these tokens across multiple layers, aligning each group with a corresponding layer. This bottom-to-top approach enhances the model’s ability to process complex visual inputs without increasing computational costs. After testing the LLaVA-1.5 and LLaVA-Next models, DeepStack shows significant performance gains across various benchmarks, particularly in high-resolution tasks, and can handle more tokens efficiently than traditional methods.

Recent advancements in LLMs like BERT, T5, …

computational costs inputs integration language language model layer machinelearningnews multimodal multimodal models multiple performance process resolution testing token tokens visual