CaMML: Context-Aware Multimodal Learner for Large Models

arXiv:2401.03149v2 Announce Type: replace
Abstract: In this work, we introduce Context-Aware MultiModal Learner (CaMML), for tuning large multimodal models (LMMs). CaMML, a lightweight module, is crafted to seamlessly integrate multimodal contextual samples into large models, thereby empowering the model to derive knowledge from analogous, domain-specific, up-to-date information and make grounded inferences. Importantly, CaMML is highly scalable and can efficiently handle lengthy multimodal context examples owing to its hierarchical design. Based on CaMML, we have developed two multimodal models, CaMML-7B and …

abstract arxiv context cs.cv domain inferences information knowledge large models large multimodal models lmms multimodal multimodal models samples type work