Deep Neural Networks Tend To Extrapolate Predictably

arXiv:2310.00873v2 Announce Type: replace
Abstract: Conventional wisdom suggests that neural network predictions tend to be unpredictable and overconfident when faced with out-of-distribution (OOD) inputs. Our work reassesses this assumption for neural networks with high-dimensional inputs. Rather than extrapolating in arbitrary ways, we observe that neural network predictions often tend towards a constant value as input data becomes increasingly OOD. Moreover, we find that this value often closely approximates the optimal constant solution (OCS), i.e., the prediction that minimizes the average …

abstract arxiv cs.lg distribution inputs network networks neural network neural networks observe predictions type value work