Learning Syntax Without Planting Trees: Understanding When and Why Transformers Generalize Hierarchically

arXiv:2404.16367v1 Announce Type: cross
Abstract: Transformers trained on natural language data have been shown to learn its hierarchical structure and generalize to sentences with unseen syntactic structures without explicitly encoding any structural bias. In this work, we investigate sources of inductive bias in transformer models and their training that could cause such generalization behavior to emerge. We extensively experiment with transformer models trained on multiple synthetic datasets and with different training objectives and show that while other objectives e.g. sequence-to-sequence …

abstract arxiv bias cs.cl cs.lg data encoding hierarchical inductive language language data learn natural natural language syntax training transformer transformer models transformers trees type understanding work