What Do Language Models Hear? Probing for Auditory Representations in Language Models

arXiv:2402.16998v1 Announce Type: cross
Abstract: This work explores whether language models encode meaningfully grounded representations of sounds of objects. We learn a linear probe that retrieves the correct text representation of an object given a snippet of audio related to that object, where the sound representation is given by a pretrained audio model. This probe is trained via a contrastive loss that pushes the language representations and sound representations of an object to be close to one another. After training, …

abstract arxiv audio cs.ai cs.cl cs.lg cs.sd eess.as encode language language models learn linear objects probe representation sound text type work