LaCViT: A Label-aware Contrastive Fine-tuning Framework for Vision Transformers

Vision Transformers (ViTs) have emerged as popular models in computer vision, demonstrating state-of-the-art performance across various tasks. This success typically follows a two-stage strategy involving pre-training on large-scale datasets using self-supervised signals, such as masked random patches, followed by fine-tuning on task-specific labeled datasets with cross-entropy loss. However, this reliance on cross-entropy loss has been identified as a limiting factor in ViTs, affecting their generalization and transferability to downstream tasks. Addressing this critical challenge, we introduce a novel Label-aware Contrastive …

art computer computer vision cross-entropy cs.ai cs.cv datasets entropy fine-tuning framework loss performance popular pre-training random reliance scale stage state strategy success tasks training transformers vision vision transformers