Brain Decodes Deep Nets

arXiv:2312.01280v2 Announce Type: replace
Abstract: We developed a tool for visualizing and analyzing large pre-trained vision models by mapping them onto the brain, thus exposing their hidden inside. Our innovation arises from a surprising usage of brain encoding: predicting brain fMRI measurements in response to images. We report two findings. First, explicit mapping between the brain and deep-network features across dimensions of space, layers, scales, and channels is crucial. This mapping method, FactorTopy, is plug-and-play for any deep-network; with it, …

abstract arxiv brain cs.cv encoding fmri hidden images innovation inside mapping report them tool type usage vision vision models