Self-Supervised Dataset Distillation for Transfer Learning

arXiv:2310.06511v3 Announce Type: replace
Abstract: Dataset distillation methods have achieved remarkable success in distilling a large dataset into a small set of representative samples. However, they are not designed to produce a distilled dataset that can be effectively used for facilitating self-supervised pre-training. To this end, we propose a novel problem of distilling an unlabeled dataset into a set of small synthetic samples for efficient self-supervised learning (SSL). We first prove that a gradient of synthetic samples with respect to …

abstract arxiv cs.lg dataset distillation however novel pre-training samples set small success training transfer transfer learning type