Self-Adapting Large Visual-Language Models to Edge Devices across Visual Modalities

arXiv:2403.04908v1 Announce Type: new
Abstract: Recent advancements in Vision-Language (VL) models have sparked interest in their deployment on edge devices, yet challenges in handling diverse visual modalities, manual annotation, and computational constraints remain. We introduce EdgeVL, a novel framework that bridges this gap by seamlessly integrating dual-modality knowledge distillation and quantization-aware contrastive learning. This approach enables the adaptation of large VL models, like CLIP, for efficient use with both RGB and non-RGB images on resource-limited devices without the need for …

abstract annotation arxiv challenges computational constraints cs.cv deployment devices distillation diverse edge edge devices framework gap knowledge language language models novel type vision visual