What explains the success of cross-modal fine-tuning with ORCA?

arXiv:2403.13537v1 Announce Type: cross
Abstract: ORCA (Shen et al., 2023) is a recent technique for cross-modal fine-tuning, i.e., applying pre-trained transformer models to modalities beyond their training data. The technique consists primarily of training an embedder and fine-tuning the embedder and model. Despite its high performance on a variety of downstream tasks, we do not understand precisely how each of these components contribute to ORCA's success. Therefore, we run a series of ablations and find that embedder training does not …

abstract arxiv beyond cs.ai cs.cl cs.cv cs.lg data fine-tuning modal orca performance success training training data transformer transformer models type