Neuro-Inspired Hierarchical Multimodal Learning

arXiv:2309.15877v3 Announce Type: replace
Abstract: Integrating and processing information from various sources or modalities are critical for obtaining a comprehensive and accurate perception of the real world. Drawing inspiration from neuroscience, we develop the Information-Theoretic Hierarchical Perception (ITHP) model, which utilizes the concept of information bottleneck. Distinct from most traditional fusion models that aim to incorporate all modalities as input, our model designates the prime modality as input, while the remaining modalities act as detectors in the information pathway. Our …

abstract aim arxiv concept cs.ai cs.lg fusion hierarchical information inspiration multimodal multimodal learning neuro neuroscience perception processing the information type world