Molecule-Space: Free Lunch in Unified Multimodal Space via Knowledge Fusion

arXiv:2405.04883v1 Announce Type: cross
Abstract: Unified multi-model representation spaces are the foundation of multimodal understanding and generation. However, the billions of model parameters and catastrophic forgetting problems make it challenging to further enhance pre-trained unified spaces. In this work, we propose Molecule-Space, an idea that treats multimodal representation spaces as "molecules", and augments pre-trained unified space by integrating knowledge from extra expert spaces via "molecules space reactions". Specifically, we introduce two kinds of basic space reactions: 1) Space Displacement Reaction …

abstract arxiv catastrophic forgetting cs.ai cs.cv cs.lg foundation free fusion however knowledge multimodal parameters representation space spaces type understanding via work