Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competition

arXiv:2402.15175v1 Announce Type: new
Abstract: Recent studies have uncovered intriguing phenomena in deep learning, such as grokking, double descent, and emergent abilities in large language models, which challenge human intuition and are crucial for a deeper understanding of neural models. In this paper, we present a comprehensive framework that provides a unified view of these three phenomena, focusing on the competition between memorization and generalization circuits. This approach, initially employed to explain grokking, is extended in our work to encompass …

abstract arxiv challenge competition cs.lg deep learning human intuition language language models large language large language models paper perspective studies type understanding view