Towards Multi-modal Transformers in Federated Learning

arXiv:2404.12467v1 Announce Type: cross
Abstract: Multi-modal transformers mark significant progress in different domains, but siloed high-quality data hinders their further improvement. To remedy this, federated learning (FL) has emerged as a promising privacy-preserving paradigm for training models without direct access to the raw data held by different clients. Despite its potential, a considerable research direction regarding the unpaired uni-modal clients and the transformer architecture in FL remains unexplored. To fill this gap, this paper explores a transfer multi-modal federated learning …

abstract access arxiv cs.cv cs.lg data domains federated learning improvement modal multi-modal paradigm privacy progress quality quality data raw research training training models transformers type