Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation

arXiv:2403.12556v1 Announce Type: new
Abstract: Previous Sign Language Translation (SLT) methods achieve superior performance by relying on gloss annotations. However, labeling high-quality glosses is a labor-intensive task, which limits the further development of SLT. Although some approaches work towards gloss-free SLT through jointly training the visual encoder and translation network, these efforts still suffer from poor performance and inefficient use of the powerful Large Language Model (LLM). Most seriously, we find that directly introducing LLM into SLT will lead to …

abstract annotations arxiv cs.cl development encoder free however labeling labor language language model language translation large language large language model performance quality through training translation type visual work