Mixed-precision Supernet Training from Vision Foundation Models using Low Rank Adapter

arXiv:2403.20080v1 Announce Type: new
Abstract: Compression of large and performant vision foundation models (VFMs) into arbitrary bit-wise operations (BitOPs) allows their deployment on various hardware. We propose to fine-tune a VFM to a mixed-precision quantized supernet. The supernet-based neural architecture search (NAS) can be adopted for this purpose, which trains a supernet, and then subnets within arbitrary hardware budgets can be extracted. However, existing methods face difficulties in optimizing the mixed-precision search space and incurring large memory costs during training. …

abstract adapter architecture arxiv compression cs.cv deployment foundation hardware low mixed mixed-precision nas neural architecture search operations precision search training type vision wise