An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits

arXiv:2212.10744v2 Announce Type: replace-cross
Abstract: Audio-visual approaches involving visual inputs have laid the foundation for recent progress in speech separation. However, the optimization of the concurrent usage of auditory and visual inputs is still an active research area. Inspired by the cortico-thalamo-cortical circuit, in which the sensory processing mechanisms of different modalities modulate one another via the non-lemniscal sensory thalamus, we propose a novel cortico-thalamo-cortical neural network (CTCNet) for audio-visual speech separation (AVSS). First, the CTCNet learns hierarchical auditory and …

arxiv audio circuits cs.cv cs.sd speech type visual