Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data

arXiv:2311.00136v4 Announce Type: replace-cross
Abstract: State-of-the-art systems neuroscience experiments yield large-scale multimodal data, and these data sets require new tools for analysis. Inspired by the success of large pretrained models in vision and language domains, we reframe the analysis of large-scale, cellular-resolution neuronal spiking data into an autoregressive spatiotemporal generation problem. Neuroformer is a multimodal, multitask generative pretrained transformer (GPT) model that is specifically designed to handle the intricacies of data in systems neuroscience. It scales linearly with feature size, …

abstract analysis art arxiv brain cellular cs.lg cs.ne data data sets domains generative language multimodal multimodal data neuroscience pretrained models pretraining q-bio.nc scale state success systems tools type vision