MIP: CLIP-based Image Reconstruction from PEFT Gradients

arXiv:2403.07901v1 Announce Type: cross
Abstract: Contrastive Language-Image Pre-training (CLIP) model, as an effective pre-trained multimodal neural network, has been widely used in distributed machine learning tasks, especially Federated Learning (FL). Typically, CLIP-based FL adopts Parameter-Efficient Fine-Tuning (PEFT) for model training, which only fine-tunes adapter parameters or soft prompts rather than the full parameters. Although PEFT is different from the traditional training mode, in this paper, we theoretically analyze that the gradients of adapters or soft prompts can still be used …

abstract adapter arxiv clip cs.cv cs.lg distributed federated learning fine-tuning image language machine machine learning multimodal network neural network parameters peft pre-training prompts tasks training type