Using an LLM to Turn Sign Spottings into Spoken Language Sentences

arXiv:2403.10434v1 Announce Type: new
Abstract: Sign Language Translation (SLT) is a challenging task that aims to generate spoken language sentences from sign language videos. In this paper, we introduce a hybrid SLT approach, Spotter+GPT, that utilizes a sign spotter and a pretrained large language model to improve SLT performance. Our method builds upon the strengths of both components. The videos are first processed by the spotter, which is trained on a linguistic sign language dataset, to identify individual signs. These …

abstract arxiv cs.cv generate gpt hybrid language language model language translation large language large language model llm paper performance spoken translation type videos