Make the Most of Your Data: Changing the Training Data Distribution to Improve In-distribution Generalization Performance

arXiv:2404.17768v1 Announce Type: new
Abstract: Can we modify the training data distribution to encourage the underlying optimization method toward finding solutions with superior generalization performance on in-distribution data? In this work, we approach this question for the first time by comparing the inductive bias of gradient descent (GD) with that of sharpness-aware minimization (SAM). By studying a two-layer CNN, we prove that SAM learns easy and difficult features more uniformly, particularly in early epochs. That is, SAM is less susceptible …

abstract arxiv bias cs.ai cs.cv cs.lg data distribution inductive optimization performance question solutions training training data type work