Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation

arXiv:2405.04164v1 Announce Type: new
Abstract: Automatic Sign Language Translation requires the integration of both computer vision and natural language processing to effectively bridge the communication gap between sign and spoken languages. However, the deficiency in large-scale training data to support sign language translation means we need to leverage resources from spoken language. We introduce, Sign2GPT, a novel framework for sign language translation that utilizes large-scale pretrained vision and language models via lightweight adapters for gloss-free sign language translation. The lightweight …

abstract and natural language processing arxiv bridge communication computer computer vision cs.cv data free gap however integration language language models language processing languages language translation large language large language models natural natural language natural language processing processing resources scale spoken support training training data translation type vision