How Tight Can PAC-Bayes be in the Small Data Regime?. (arXiv:2106.03542v4 [stat.ML] UPDATED)

In this paper, we investigate the question: Given a small number of
datapoints, for example N = 30, how tight can PAC-Bayes and test set bounds be
made? For such small datasets, test set bounds adversely affect generalisation
performance by withholding data from the training procedure. In this setting,
PAC-Bayes bounds are especially attractive, due to their ability to use all the
data to simultaneously learn a posterior and bound its generalisation risk. We
focus on the case of i.i.d. …

arxiv bayes data ml small small data