Neural Networks Make Approximately Independent Errors Over Repeated Training

arXiv:2304.01910v2 Announce Type: replace
Abstract: Typical neural network trainings have substantial variance in test-set performance between repeated runs, impeding hyperparameter comparison and training reproducibility. In this work we present the following results towards understanding this variation. (1) Despite having significant variance on their test-sets, we demonstrate that standard CIFAR-10 and ImageNet trainings have little variance in performance on the underlying test-distributions from which their test-sets are sampled. (2) We show that these trainings make approximately independent errors on their test-sets. …

abstract arxiv cifar-10 comparison cs.lg errors hyperparameter imagenet independent network networks neural network neural networks performance reproducibility results set standard test training type understanding variance variation work