To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets

arXiv:2310.13061v2 Announce Type: replace
Abstract: Robust generalization is a major challenge in deep learning, particularly when the number of trainable parameters is very large. In general, it is very difficult to know if the network has memorized a particular set of examples or understood the underlying rule (or both). Motivated by this challenge, we study an interpretable model where generalizing representations are understood analytically, and are easily distinguishable from the memorizing ones. Namely, we consider multi-layer perceptron (MLP) and Transformer …

abstract arxiv challenge cond-mat.dis-nn cs.lg datasets deep learning examples general grok major network parameters robust set stat.ml type