Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgetting

arXiv:2404.17245v1 Announce Type: new
Abstract: Artificial neural networks often suffer from catastrophic forgetting, where learning new concepts leads to a complete loss of previously acquired knowledge. We observe that this issue is particularly magnified in vision transformers (ViTs), where post-pre-training and fine-tuning on new tasks can significantly degrade the model's original general abilities. For instance, a DINO ViT-Base/16 pre-trained on ImageNet-1k loses over 70% accuracy on ImageNet-1k after just 10 iterations of fine-tuning on CIFAR-100. Overcoming this stability-plasticity dilemma is …

abstract acquired artificial artificial neural networks arxiv catastrophic forgetting concepts cs.cv fine-tuning issue knowledge leads loss networks neural networks observe pre-training tasks training transformers type vision vision transformers