Supervised Fine-tuning in turn Improves Visual Foundation Models

arXiv:2401.10222v2 Announce Type: replace
Abstract: Image-text training like CLIP has dominated the pretraining of vision foundation models in recent years. Subsequent efforts have been made to introduce region-level visual learning into CLIP's pretraining but face scalability challenges due to the lack of large-scale region-level datasets. Drawing inspiration from supervised fine-tuning (SFT) in natural language processing such as instruction tuning, we explore the potential of fine-grained SFT in enhancing the generation of vision foundation models after their pretraining. Thus a two-stage …

abstract arxiv challenges clip cs.ai cs.cv datasets face fine-tuning foundation image inspiration pretraining scalability scale sft supervised fine-tuning text training type vision visual