Modality-Agnostic fMRI Decoding of Vision and Language

arXiv:2403.11771v1 Announce Type: new
Abstract: Previous studies have shown that it is possible to map brain activation data of subjects viewing images onto the feature representation space of not only vision models (modality-specific decoding) but also language models (cross-modal decoding). In this work, we introduce and use a new large-scale fMRI dataset (~8,500 trials per subject) of people watching both images and text descriptions of such images. This novel dataset enables the development of modality-agnostic decoders: a single decoder that …

abstract arxiv brain cs.cl cs.cv data dataset decoding feature fmri images language language models map modal representation scale space studies type vision vision models work