Out-of-Distribution Generalization in Algorithmic Reasoning Through Curriculum Learning. (arXiv:2210.03275v1 [cs.LG])

Out-of-distribution generalization (OODG) is a longstanding challenge for
neural networks, and is quite apparent in tasks with well-defined variables and
rules, where explicit use of the rules can solve problems independently of the
particular values of the variables. Large transformer-based language models
have pushed the boundaries on how well neural networks can generalize to novel
inputs, but their complexity obfuscates they achieve such robustness. As a step
toward understanding how transformer-based systems generalize, we explore the
question of OODG in …

arxiv curriculum curriculum learning distribution reasoning