BAT: Learning to Reason about Spatial Sounds with Large Language Models

Spatial sound reasoning is a fundamental human skill, enabling us to navigate and interpret our surroundings based on sound. In this paper we present BAT, which combines the spatial sound perception ability of a binaural acoustic scene analysis model with the natural language reasoning capabilities of a large language model (LLM) to replicate this innate ability. To address the lack of existing datasets of in-the-wild spatial sounds, we synthesized a binaural audio dataset using AudioSet and SoundSpaces 2.0. Next, we …

analysis bat capabilities cs.ai cs.cl cs.sd eess.as enabling human language language models large language large language models natural natural language paper perception reason reasoning sound spatial