Self-supervised Dataset Distillation: A Good Compression Is All You Need

arXiv:2404.07976v1 Announce Type: new
Abstract: Dataset distillation aims to compress information from a large-scale original dataset to a new compact dataset while striving to preserve the utmost degree of the original data informational essence. Previous studies have predominantly concentrated on aligning the intermediate statistics between the original and distilled data, such as weight trajectory, features, gradient, BatchNorm, etc. In this work, we consider addressing this task through the new lens of model informativeness in the compression stage on the original …

arxiv compression cs.ai cs.cv dataset distillation good type