UMBRAE: Unified Multimodal Decoding of Brain Signals

arXiv:2404.07202v1 Announce Type: new
Abstract: We address prevailing challenges of the brain-powered research, departing from the observation that the literature hardly recover accurate spatial information and require subject-specific models. To address these challenges, we propose UMBRAE, a unified multimodal decoding of brain signals. First, to extract instance-level conceptual and spatial details from neural signals, we introduce an efficient universal brain encoder for multimodal-brain alignment and recover object descriptions at multiple levels of granularity from subsequent multimodal large language model (MLLM). …

arxiv brain brain signals cs.ai cs.cl cs.cv decoding multimodal type