Language Models are Free Boosters for Biomedical Imaging Tasks

arXiv:2403.17343v1 Announce Type: cross
Abstract: In this study, we uncover the unexpected efficacy of residual-based large language models (LLMs) as part of encoders for biomedical imaging tasks, a domain traditionally devoid of language or textual data. The approach diverges from established methodologies by utilizing a frozen transformer block, extracted from pre-trained LLMs, as an innovative encoder layer for the direct processing of visual tokens. This strategy represents a significant departure from the standard multi-modal vision-language frameworks, which typically hinge on …

abstract arxiv biomedical block cs.cl cs.cv cs.lg data domain free imaging language language models large language large language models llms part residual study tasks textual transformer type