Early Period of Training Impacts Out-of-Distribution Generalization

arXiv:2403.15210v1 Announce Type: new
Abstract: Prior research has found that differences in the early period of neural network training significantly impact the performance of in-distribution (ID) tasks. However, neural networks are often sensitive to out-of-distribution (OOD) data, making them less reliable in downstream applications. Yet, the impact of the early training period on OOD generalization remains understudied due to its complexity and lack of effective analytical methodologies. In this work, we investigate the relationship between learning dynamics and OOD generalization …

abstract applications arxiv cs.lg data differences distribution found however impact impacts making network networks network training neural network neural networks performance prior research tasks them training type