Integrating kNN with Foundation Models for Adaptable and Privacy-Aware Image Classification

arXiv:2402.12500v1 Announce Type: cross
Abstract: Traditional deep learning models implicity encode knowledge limiting their transparency and ability to adapt to data changes. Yet, this adaptability is vital for addressing user data privacy concerns. We address this limitation by storing embeddings of the underlying training data independently of the model weights, enabling dynamic data modifications without retraining. Specifically, our approach integrates the $k$-Nearest Neighbor ($k$-NN) classifier with a vision-based foundation model, pre-trained self-supervised on natural images, enhancing interpretability and adaptability. We …

arxiv classification cs.cv cs.lg eess.iv foundation image knn privacy type