Language Generation in the Limit

arXiv:2404.06757v1 Announce Type: cross
Abstract: Although current large language models are complex, the most basic specifications of the underlying language generation problem itself are simple to state: given a finite set of training samples from an unknown language, produce valid new strings from the language that don't already appear in the training data. Here we ask what we can conclude about language generation using only this specification, without further assumptions. In particular, suppose that an adversary enumerates the strings of …

abstract arxiv basic cs.ai cs.cl cs.ds cs.lg current data language language generation language models large language large language models samples set simple state strings training training data type